Nov 03 10:28:17.490868 ip-10-0-138-233 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 10:28:17.869454 ip-10-0-138-233 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:28:17.869454 ip-10-0-138-233 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 10:28:17.869454 ip-10-0-138-233 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:28:17.869454 ip-10-0-138-233 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:28:17.874595 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874477    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:17.874595 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874588    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:17.874595 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874592    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:17.874595 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874595    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:17.874595 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874598    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:17.874595 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874600    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:17.874595 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874603    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874606    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874609    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874611    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874614    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874618    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874620    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874632    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874635    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874637    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874639    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874642    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874645    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874648    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874650    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874653    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874656    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874658    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874661    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874663    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:17.874839 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874666    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874670    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874675    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874678    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874681    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874684    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874687    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874693    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874695    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874698    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874701    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874703    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874706    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874709    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874712    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874715    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874718    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874721    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874724    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:17.875300 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874727    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874729    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874732    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874734    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874737    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874740    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874742    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874745    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874747    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874750    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874752    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874755    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874757    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874759    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874762    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874764    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874767    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874769    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874772    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874774    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:17.875763 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874782    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874784    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874787    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874790    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874792    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874795    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874799    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874801    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874804    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874806    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874809    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874811    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874814    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874817    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874821    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874826    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874829    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874832    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874835    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:17.876243 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874838    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874841    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874843    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874846    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874849    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874851    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874854    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874857    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874859    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874862    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874864    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874867    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874869    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874873    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874881    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874883    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874886    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874888    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874891    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874895    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:17.876708 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874897    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.874900    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875733    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875741    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875744    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875747    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875750    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875753    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875755    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875758    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875760    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875764    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875766    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875769    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875772    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875775    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875777    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875780    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875782    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875785    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:17.877195 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875787    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875790    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875792    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875795    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875797    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875800    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875802    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875805    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875809    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875813    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875816    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875820    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875825    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875828    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875831    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875833    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875836    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875838    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875841    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:17.877720 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875844    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875846    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875849    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875851    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875853    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875856    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875858    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875861    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875864    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875866    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875869    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875872    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875874    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875877    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875879    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875882    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875884    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875887    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875889    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875897    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:17.878166 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875900    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875903    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875905    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875908    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875913    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875916    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875918    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875921    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875923    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875926    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875928    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875931    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875934    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875936    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875939    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875941    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875944    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875947    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875949    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875952    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:17.878641 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875954    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875957    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875959    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875962    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875964    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875967    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875969    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875972    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875974    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875977    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875979    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875981    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875984    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875986    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875990    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875993    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875997    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.875999    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876002    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:17.879146 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876004    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876007    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876009    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876012    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876015    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876017    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876020    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876022    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876026    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876028    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876478    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876483    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876486    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876488    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876491    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876494    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876496    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876499    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876501    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876504    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:17.879600 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876506    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876509    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876511    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876514    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876516    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876519    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876521    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876525    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876528    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876533    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876535    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876538    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876541    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876543    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876546    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876548    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876551    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876553    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876556    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876558    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:17.880073 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876561    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876563    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876566    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876585    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876587    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876590    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876593    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876595    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876599    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876603    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876605    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876608    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876610    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876613    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876615    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876618    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876620    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876623    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876625    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:17.880529 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876628    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876631    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876635    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876638    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876640    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876643    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876645    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876648    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876650    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876653    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876655    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876658    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876660    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876663    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876665    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876668    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876671    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876673    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876676    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:17.880986 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876678    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876681    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876683    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876686    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876689    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876691    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876694    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876696    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876699    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876702    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876705    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876707    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876710    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876712    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876715    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876721    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876723    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876726    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876729    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876731    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:17.881435 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876734    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876736    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876739    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876741    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876744    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876746    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876749    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876751    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876754    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876756    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876759    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876763    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876767    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876771    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876774    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876777    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876781    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.876784    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877616    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877626    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 10:28:17.881921 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877632    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877636    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877640    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877644    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877648    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877652    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877655    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877661    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877666    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877670    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877673    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877676    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877678    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877681    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877684    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877687    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877692    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877694    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877697    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877700    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877705    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877709    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877712    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 10:28:17.882389 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877715    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877718    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877721    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877724    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877727    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877730    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877734    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877738    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877741    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877744    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877747    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877750    2576 flags.go:64] FLAG: --enable-server="true"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877753    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877757    2576 flags.go:64] FLAG: --event-burst="100"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877760    2576 flags.go:64] FLAG: --event-qps="50"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877763    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877767    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877770    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877775    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877778    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877781    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877784    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877787    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877789    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877792    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 10:28:17.882955 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877795    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877798    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877801    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877804    2576 flags.go:64] FLAG: --feature-gates=""
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877808    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877811    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877813    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877816    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877819    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877822    2576 flags.go:64] FLAG: --help="false"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877825    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877828    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877831    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877834    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877837    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877841    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877843    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877846    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877849    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877852    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877855    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877858    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877860    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877863    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 10:28:17.883622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877866    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877869    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877873    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877876    2576 flags.go:64] FLAG: --lock-file=""
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877878    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877881    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877884    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877889    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877892    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877895    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877898    2576 flags.go:64] FLAG: --logging-format="text"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877901    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877904    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877907    2576 flags.go:64] FLAG: --manifest-url=""
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877910    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877914    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877917    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877921    2576 flags.go:64] FLAG: --max-pods="110"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877923    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877926    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877929    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877932    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877935    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877937    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877940    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 10:28:17.884333 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877947    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877950    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877953    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877956    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877958    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877961    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877964    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877967    2576 flags.go:64] FLAG: --port="10250"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877970    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877973    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02ea222e3a0053047"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877976    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877979    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877982    2576 flags.go:64] FLAG: --register-node="true"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877984    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877988    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877991    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877993    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.877996    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878000    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878003    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878006    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878009    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878011    2576 flags.go:64] FLAG: --runonce="false"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878014    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878017    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878020    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 10:28:17.884957 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878023    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878026    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878028    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878031    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878034    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878037    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878039    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878042    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878045    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878048    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878051    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878053    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878058    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878061    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878064    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878071    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878074    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878077    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878080    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878083    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878086    2576 flags.go:64] FLAG: --v="2"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878090    2576 flags.go:64] FLAG: --version="false"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878095    2576 flags.go:64] FLAG: --vmodule=""
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878099    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878102    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 10:28:17.885672 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878232    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878235    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878238    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878241    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878245    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878247    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878250    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878252    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878255    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878257    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878260    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878263    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878265    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878269    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878272    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878275    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878277    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878280    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878284    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:17.886676 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878287    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878290    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878292    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878295    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878297    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878300    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878303    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878305    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878308    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878311    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878313    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878316    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878318    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878321    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878323    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878326    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878328    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878330    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878333    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878336    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:17.887193 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878338    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878341    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878344    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878346    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878349    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878351    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878354    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878356    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878359    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878361    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878364    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878370    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878372    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878375    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878378    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878380    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878383    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878385    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878388    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878391    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:17.887678 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878393    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878396    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878399    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878402    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878404    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878407    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878409    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878412    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878414    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878416    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878419    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878421    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878424    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878426    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878429    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878431    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878434    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878436    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878439    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:17.888143 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878441    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878443    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878446    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878448    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878453    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878456    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878458    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878461    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878463    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878466    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878468    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878471    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878473    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878476    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878478    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878481    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878484    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878486    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878488    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878491    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:17.888683 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878493    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878496    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878498    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878501    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878503    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878506    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878509    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878511    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.878536    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878583    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878588    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878592    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878594    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878597    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878600    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:17.889140 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878602    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878608    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878612    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878615    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878618    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878621    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878624    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878626    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878629    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878631    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878634    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878639    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878643    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878646    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878649    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878652    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878654    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878657    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878659    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:17.889502 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878662    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878664    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878667    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878669    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878672    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878674    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878677    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878680    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878682    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878685    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878687    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878690    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878692    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878695    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878699    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878703    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878705    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878708    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878711    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878714    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:17.889972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878716    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878719    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878721    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878724    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878726    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878730    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878732    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878735    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878738    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878740    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878743    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878745    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878748    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878750    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878753    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878755    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878757    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878760    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878762    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:17.890434 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878765    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878768    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878770    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878773    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878775    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878777    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878780    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878784    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878788    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878790    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878793    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878795    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878798    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878800    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878803    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878805    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878808    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878811    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878813    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:17.890887 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878816    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878818    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878821    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878824    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878826    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878828    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878831    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878833    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878836    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878838    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878841    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878843    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878846    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878848    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878850    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878853    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878855    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878858    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878860    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878863    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:17.891323 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878867    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878870    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.878873    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.887607    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.887621    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.888058    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.888065    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888123    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888129    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888132    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888135    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888140    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888144    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888147    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888151    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888154    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888158    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888161    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:17.891813 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888164    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888166    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888169    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888171    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888174    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888177    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888182    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888186    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888191    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888195    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888200    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888204    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888207    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888210    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888213    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888216    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888220    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888222    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888225    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:17.892229 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888227    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888230    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888232    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888235    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888237    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888240    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888243    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888245    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888248    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888250    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888253    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888255    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888258    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888260    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888263    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888265    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888267    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888270    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888272    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888275    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:17.892793 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888277    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888280    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888283    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888285    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888288    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888291    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888293    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888296    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888298    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888301    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888303    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888306    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888309    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888311    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888314    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888317    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888319    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888322    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888325    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888327    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:17.893269 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888329    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888332    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888334    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888337    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888339    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888342    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888345    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888347    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888350    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888353    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888356    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888358    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888361    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888363    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888366    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888368    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888371    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888373    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888376    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888379    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:17.893779 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888381    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888384    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888386    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888389    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888391    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888394    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888396    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888399    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888401    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888404    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888406    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888409    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888411    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888413    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888416    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888418    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:17.894251 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.888446    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888641    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888648    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888652    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888656    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888658    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888662    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888665    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888667    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888670    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888672    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888675    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888677    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888681    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888686    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888690    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888693    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888696    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888699    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:17.894633 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888702    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888705    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888708    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888710    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888713    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888715    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888718    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888720    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888723    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888725    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888728    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888730    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888733    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888735    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888737    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888740    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888742    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888745    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888747    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888750    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:17.895067 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888753    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888755    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888758    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888760    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888763    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888765    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888768    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888770    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888773    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888776    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888779    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888781    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888785    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888788    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888791    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888794    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888797    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888800    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888802    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:17.895535 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888804    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888807    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888809    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888812    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888814    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888817    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888819    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888822    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888825    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888827    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888830    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888832    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888835    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888837    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888840    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888843    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888845    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888848    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888850    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888853    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:17.895989 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888855    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888858    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888860    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888863    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888865    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888868    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888870    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888873    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888875    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888878    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888880    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888883    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888886    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888888    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888891    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888893    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888895    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888898    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888900    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:17.896463 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888903    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888905    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888908    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888910    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888913    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888915    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888917    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888920    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888923    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888925    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.888948    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888979    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888982    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888987    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888990    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888993    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:17.896909 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.888998    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889000    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889003    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889005    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889008    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889010    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889013    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889015    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889018    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889022    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889026    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889029    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889033    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889035    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889038    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889040    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889043    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889046    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889048    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:17.897296 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889051    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889053    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889056    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889058    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889061    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889063    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889066    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889069    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889072    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889074    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889078    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889081    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889083    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889087    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889090    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889092    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889095    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889097    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889100    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889102    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:17.897754 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889105    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889108    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889110    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889113    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889115    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889118    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889120    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889123    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889127    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889130    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889133    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889135    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889138    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889140    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889143    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889153    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889156    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889159    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889162    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:17.898205 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889165    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889167    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889170    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889174    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889177    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889179    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889183    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889186    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889188    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889191    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889194    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889196    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889198    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889201    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889204    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889207    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889210    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889212    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889215    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889217    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:17.898649 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889220    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889222    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889225    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889227    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889230    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889233    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889235    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889238    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889240    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889243    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889245    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889248    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889250    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889253    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889256    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889259    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889263    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889272    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:17.899116 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889277    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889281    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889286    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:17.889290    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.889340    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.891228    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.893053    2576 server.go:1014] "Starting client certificate rotation"
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.893155    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.893858    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.895037    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:28:17.899587 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.898419    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:28:17.909142 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.909121    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 03 10:28:17.913255 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.913239    2576 log.go:25] "Validated CRI v1 image API"
Nov 03 10:28:17.914042 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.914016    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 10:28:17.917890 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.917871    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a5f97d21-a382-4edb-9d93-858f82c22150:/dev/nvme0n1p3 c3583c8b-86ad-4901-bdaa-949b653389d0:/dev/nvme0n1p4]
Nov 03 10:28:17.917956 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.917890    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 10:28:17.922826 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.922807    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 10:28:17.924147 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.924041    2576 manager.go:217] Machine: {Timestamp:2026-11-03 10:28:17.922955537 +0000 UTC m=+0.320915159 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099173 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27202688a9a1e27e73ef4d0b7cddf5 SystemUUID:ec272026-88a9-a1e2-7e73-ef4d0b7cddf5 BootID:f9dbc59b-a26e-44fb-9a86-88fc687e9664 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:04:80:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:04:80:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:2f:a4:5f:85:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 10:28:17.924147 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.924147    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 10:28:17.924272 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.924259    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 10:28:17.925501 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.925473    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 10:28:17.925700 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.925502    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-233.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 10:28:17.925790 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.925707    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 10:28:17.925790 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.925720    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 10:28:17.925790 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.925734    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 10:28:17.926533 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.926522    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 10:28:17.927325 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.927313    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 10:28:17.931856 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.931825    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 10:28:17.932224 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.932211    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 10:28:17.932297 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.932283    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 10:28:17.932344 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.932315    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 10:28:17.932344 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.932330    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 10:28:17.932429 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.932368    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 03 10:28:17.932429 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.932379    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 10:28:17.932972 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.932958    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 10:28:17.935700 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.935677    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 10:28:17.939057 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.939044    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 10:28:17.939384 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.939369    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ffd4t"
Nov 03 10:28:17.939890 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.939879    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 10:28:17.939939 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.939909    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 10:28:17.940515 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940505    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 10:28:17.940549 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940522    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 10:28:17.940549 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940528    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 10:28:17.940549 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940533    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 10:28:17.940549 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940539    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 10:28:17.940549 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940545    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 10:28:17.940969 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940961    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 10:28:17.940997 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940970    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 10:28:17.940997 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940977    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 10:28:17.940997 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940983    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 10:28:17.940997 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940991    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 10:28:17.941371 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.940999    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 10:28:17.942278 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.942269    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 10:28:17.942278 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.942278    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 10:28:17.945336 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.945315    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-233.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 10:28:17.945410 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.945366    2576 server.go:1263] "Started kubelet"
Nov 03 10:28:17.945512 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.945461    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 10:28:17.945564 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.945530    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 10:28:17.945626 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.945529    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 10:28:17.946126 ip-10-0-138-233 systemd[1]: Started Kubernetes Kubelet.
Nov 03 10:28:17.946825 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.946596    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 10:28:17.947250 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.947182    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ffd4t"
Nov 03 10:28:17.947509 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:17.947488    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-233.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 10:28:17.947615 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:17.947561    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 10:28:17.948630 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.948615    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 10:28:17.951498 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:17.951476    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 10:28:17.952054 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.952028    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 10:28:17.952428 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.952413    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 10:28:17.952607 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:17.952587    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:17.953027 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953009    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 10:28:17.953092 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953027    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 10:28:17.953092 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953045    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 10:28:17.953172 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953160    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 10:28:17.953222 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953172    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 10:28:17.953314 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953297    2576 factory.go:55] Registering systemd factory
Nov 03 10:28:17.953314 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953314    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 03 10:28:17.953554 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953537    2576 factory.go:153] Registering CRI-O factory
Nov 03 10:28:17.953640 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953558    2576 factory.go:223] Registration of the crio container factory successfully
Nov 03 10:28:17.953640 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953620    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 10:28:17.953640 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953636    2576 factory.go:103] Registering Raw factory
Nov 03 10:28:17.953838 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.953646    2576 manager.go:1199] Started watching for new ooms in manager
Nov 03 10:28:17.954429 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.954413    2576 manager.go:319] Starting recovery of all containers
Nov 03 10:28:17.954920 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:17.953952    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-233.us-east-2.compute.internal.18e48230db0c0eb6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-233.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-233.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 10:28:17.945341622 +0000 UTC m=+0.343301250,LastTimestamp:2026-11-03 10:28:17.945341622 +0000 UTC m=+0.343301250,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-233.us-east-2.compute.internal,}"
Nov 03 10:28:17.960535 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.960513    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:17.962875 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:17.962841    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-233.us-east-2.compute.internal\" not found" node="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:17.966113 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.966096    2576 manager.go:324] Recovery completed
Nov 03 10:28:17.970337 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.970324    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:17.971977 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.971962    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:17.972050 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.971994    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:17.972050 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.972006    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:17.972497 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.972483    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 03 10:28:17.972497 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.972493    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:17.972648 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.972494    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 10:28:17.972648 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.972560    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 10:28:17.973194 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.973175    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:17.973272 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.973205    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:17.973272 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.973219    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:17.974609 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.974597    2576 policy_none.go:50] "Start"
Nov 03 10:28:17.974647 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.974614    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 10:28:17.974647 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.974625    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 10:28:17.976513 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:17.976502    2576 policy_none.go:44] "Start"
Nov 03 10:28:18.016231 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.016106    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 10:28:18.020425 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.020413    2576 manager.go:341] "Starting Device Plugin manager"
Nov 03 10:28:18.020502 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.020441    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 10:28:18.020502 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.020449    2576 server.go:87] "Starting device plugin registration server"
Nov 03 10:28:18.020750 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.020740    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 10:28:18.020798 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.020776    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 10:28:18.020866 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.020852    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 10:28:18.020951 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.020941    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 10:28:18.021001 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.020954    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 10:28:18.021528 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.021506    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 10:28:18.021628 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.021554    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.111385 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.111360    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 10:28:18.113384 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.112663    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 10:28:18.113384 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.112688    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 10:28:18.113384 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.112708    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 10:28:18.113384 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.112746    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 10:28:18.116108 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.116092    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:18.121811 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.121772    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:18.122526 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.122511    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:18.122599 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.122540    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:18.122599 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.122558    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:18.122599 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.122599    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.128979 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.128965    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.129020 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.128985    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-233.us-east-2.compute.internal\": node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.147178 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.147157    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.213401 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.213378    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"]
Nov 03 10:28:18.213479 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.213437    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:18.214305 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.214291    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:18.214376 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.214319    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:18.214376 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.214329    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:18.216376 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.216363    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:18.216514 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.216499    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.216558 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.216530    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:18.217194 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.217169    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:18.217194 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.217195    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:18.217319 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.217205    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:18.217319 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.217206    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:18.217319 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.217234    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:18.217319 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.217245    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:18.219255 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.219239    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.219339 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.219269    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:18.226897 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.226882    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:18.226990 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.226911    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:18.226990 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.226924    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:18.247466 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.247450    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.255188 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.255172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.255269 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.255196    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.255269 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.255220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.257818 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.257797    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" node="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.259793 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.259779    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" node="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.348333 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.348313    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.355674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.355655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.355731 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.355680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.355731 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.355705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.355797 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.355749    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.355797 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.355764    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.355860 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.355802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.449087 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.449024    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.549532 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.549496    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.559950 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.559929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.562104 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.562084    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:18.650128 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.650095    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.750688 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.750615    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.851098 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.851073    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.892615 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.892566    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 10:28:18.893025 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.892726    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 10:28:18.893025 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.892751    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 10:28:18.951562 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:18.951541    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:28:18.953234 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.953169    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 10:23:18 +0000 UTC" deadline="2028-08-02 07:19:15.377302006 +0000 UTC"
Nov 03 10:28:18.953283 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:18.953235    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15308h50m56.42407058s"
Nov 03 10:28:19.010064 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.010013    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:19.052805 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.052786    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:19.063011 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.062994    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:28:19.064806 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.064794    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:28:19.070473 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.070453    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:28:19.130267 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.130246    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:19.268162 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:19.268099    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb181debb656c62bb70f1078166094d.slice/crio-299b108faa17dcf3a53dedb8a4ce351eb7ba8cbad39ad2c039d631c363be1e0a WatchSource:0}: Error finding container 299b108faa17dcf3a53dedb8a4ce351eb7ba8cbad39ad2c039d631c363be1e0a: Status 404 returned error can't find the container with id 299b108faa17dcf3a53dedb8a4ce351eb7ba8cbad39ad2c039d631c363be1e0a
Nov 03 10:28:19.268302 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:19.268283    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0906d8be50774c4e8328161c8119d16.slice/crio-46e383e26addea4fc9566f042bf9f8e95e05d495fa7408e93df367a4ac593630 WatchSource:0}: Error finding container 46e383e26addea4fc9566f042bf9f8e95e05d495fa7408e93df367a4ac593630: Status 404 returned error can't find the container with id 46e383e26addea4fc9566f042bf9f8e95e05d495fa7408e93df367a4ac593630
Nov 03 10:28:19.273834 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.273817    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:28:19.800314 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.800239    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:19.934937 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.934899    2576 apiserver.go:52] "Watching apiserver"
Nov 03 10:28:19.943458 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.943317    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 10:28:19.943784 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.943760    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-cxvr7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-j6sfm","kube-system/konnectivity-agent-8v4lr","openshift-multus/network-metrics-daemon-4qhrr","openshift-network-operator/iptables-alerter-6kr4v","kube-system/global-pull-secret-syncer-zwj2x","openshift-multus/multus-additional-cni-plugins-jtzkm","openshift-cluster-node-tuning-operator/tuned-fqldz","openshift-multus/multus-pf78w","openshift-network-diagnostics/network-check-target-brkhs","kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n","openshift-dns/node-resolver-9bmpc"]
Nov 03 10:28:19.946479 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.946449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:19.949257 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.949236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:19.950934 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.950894    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 10:28:19.950934 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.950913    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 10:28:19.950934 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.950898    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 10:28:19.951227 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.951048    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4c7q2\""
Nov 03 10:28:19.951736 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.951714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.952237 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.952216    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 10:28:19.952542 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.952519    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nxhw\""
Nov 03 10:28:19.952628 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.952554    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 10:28:19.952628 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.952616    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 10:28:19.954558 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.954539    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 10:28:19.955632 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.955497    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 10:28:19.955632 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.955530    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 10:28:19.955632 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.955555    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 10:28:19.955632 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.955581    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 10:28:19.955901 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.955612    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xjpbn\""
Nov 03 10:28:19.955901 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.955711    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 10:28:19.956341 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.956309    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 10:28:19.956428 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.956349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:19.956491 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:19.956417    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:19.958510 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.958492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:19.962065 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.961494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:19.964398 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.964380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:19.964498 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:19.964450    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-env-overrides\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-var-lib-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-etc-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-ovn\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-host\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-systemd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-script-lib\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-registration-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-device-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-slash\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-node-log\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-config\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6v8d\" (UniqueName: \"kubernetes.io/projected/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-kube-api-access-d6v8d\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.966933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-systemd-units\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctqm\" (UniqueName: \"kubernetes.io/projected/b96ecc89-8209-4f18-bc24-e96628147236-kube-api-access-9ctqm\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-kubelet\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-netns\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.966990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-netd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9jxn\" (UniqueName: \"kubernetes.io/projected/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-kube-api-access-n9jxn\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-socket-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-sys-fs\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovn-node-metrics-cert\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-serviceca\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-log-socket\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.967539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-bin\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:19.968450 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.967740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:19.970677 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.970051    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:19.970677 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:19.970162    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:19.973690 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.973334    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pf78w"
Nov 03 10:28:19.976233 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.976212    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:19.978681 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.978660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:19.989065 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.989047    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 10:28:19.989170 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.989103    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t8gfp"
Nov 03 10:28:19.994068 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:19.994050    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t8gfp"
Nov 03 10:28:20.054020 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.053967    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 10:28:20.060243 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060213    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:28:20.060381 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060364    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 10:28:20.060466 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060437    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 10:28:20.060541 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060526    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 10:28:20.060626 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060595    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 10:28:20.060695 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060681    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 10:28:20.060785 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060766    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 10:28:20.060866 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060842    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:28:20.060945 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060926    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 10:28:20.060999 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.060989    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 10:28:20.061072 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.061051    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 10:28:20.061160 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.061147    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 10:28:20.061234 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.061222    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 10:28:20.061365 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.061352    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 10:28:20.061781 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.061762    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wbpkj\""
Nov 03 10:28:20.061899 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.061883    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zcr5r\""
Nov 03 10:28:20.061962 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.061946    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-khtn4\""
Nov 03 10:28:20.062102 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.062020    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 10:28:20.062184 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.062115    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5hs5j\""
Nov 03 10:28:20.062228 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.062182    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jkrmv\""
Nov 03 10:28:20.062265 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.062238    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-trm8g\""
Nov 03 10:28:20.067338 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-daemon-config\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.067449 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-kubelet\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.067449 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-netns\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.067449 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.067449 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.067449 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0174417f-2cb0-4933-b9cc-7b044e2e8254-agent-certs\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jxn\" (UniqueName: \"kubernetes.io/projected/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-kube-api-access-n9jxn\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-kubernetes\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067554    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-run\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-cnibin\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-k8s-cni-cncf-io\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-bin\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysconfig\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.067734 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067723    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-kubelet\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.068236 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-netns\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.068338 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.068314    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-bin\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.068400 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.067833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.068614 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.068585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-systemd\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.071685 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.068765    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.071872 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.071691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-var-lib-kubelet\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.071872 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.071748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/316f1ae7-9bef-4691-a73d-e79f01fdea14-host-slash\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.072002 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.071897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/565f923e-a384-496e-be31-f99ddaec54ce-hosts-file\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.072002 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.071937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-env-overrides\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jstbl\" (UniqueName: \"kubernetes.io/projected/40cc057e-3d16-464d-8b2c-7a15d01238c1-kube-api-access-jstbl\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-cnibin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-host\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-host\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-systemd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-script-lib\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-tuned\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-cni-binary-copy\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4c6f\" (UniqueName: \"kubernetes.io/projected/565f923e-a384-496e-be31-f99ddaec54ce-kube-api-access-h4c6f\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-device-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-config\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072783    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-bin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072898    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-multus\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.073673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-conf\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmv4\" (UniqueName: \"kubernetes.io/projected/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-kube-api-access-cbmv4\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.072996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrz7t\" (UniqueName: \"kubernetes.io/projected/5e735188-d000-4d3e-9e65-f3320352f3b5-kube-api-access-jrz7t\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctqm\" (UniqueName: \"kubernetes.io/projected/b96ecc89-8209-4f18-bc24-e96628147236-kube-api-access-9ctqm\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-netd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-socket-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-sys-fs\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovn-node-metrics-cert\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-serviceca\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-log-socket\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwp9g\" (UniqueName: \"kubernetes.io/projected/316f1ae7-9bef-4691-a73d-e79f01fdea14-kube-api-access-rwp9g\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-system-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-sys\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-netns\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.075986 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-hostroot\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-multus-certs\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565f923e-a384-496e-be31-f99ddaec54ce-tmp-dir\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0174417f-2cb0-4933-b9cc-7b044e2e8254-konnectivity-ca\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-var-lib-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-env-overrides\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-etc-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-ovn\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-lib-modules\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-socket-dir-parent\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-tmp\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/316f1ae7-9bef-4691-a73d-e79f01fdea14-iptables-alerter-script\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-conf-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-dbus\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.076801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-registration-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073923    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-socket-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-slash\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073970    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-slash\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.073982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-node-log\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v8d\" (UniqueName: \"kubernetes.io/projected/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-kube-api-access-d6v8d\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074021    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-log-socket\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-etc-kubernetes\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-sys-fs\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074146    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-device-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074159    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-var-lib-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074248    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-registration-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074614    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-config\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074691    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-host\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.074701    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.074722    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.074735    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mlg5g for pod openshift-network-diagnostics/network-check-target-brkhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:20.077674 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074746    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-netd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.074795    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g podName:7075e9e0-fca9-44ba-82bf-e93c702b576d nodeName:}" failed. No retries permitted until 2026-11-03 10:28:20.574774999 +0000 UTC m=+2.972734617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mlg5g" (UniqueName: "kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g") pod "network-check-target-brkhs" (UID: "7075e9e0-fca9-44ba-82bf-e93c702b576d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074813    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074853    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-node-log\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.074874    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-script-lib\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075161    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075302    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-serviceca\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075358    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-systemd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075406    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075522    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-kubelet-config\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-systemd-units\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-modprobe-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfqf\" (UniqueName: \"kubernetes.io/projected/cd31aaf2-4824-4357-818f-aaa8663c3917-kube-api-access-nbfqf\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.075729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-kubelet\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.078543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.076200    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-systemd-units\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.079380 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.076403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-etc-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.079380 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.076607    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-ovn\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.079799 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.079754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovn-node-metrics-cert\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.081611 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.081573    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jxn\" (UniqueName: \"kubernetes.io/projected/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-kube-api-access-n9jxn\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:20.082413 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.082389    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctqm\" (UniqueName: \"kubernetes.io/projected/b96ecc89-8209-4f18-bc24-e96628147236-kube-api-access-9ctqm\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.090343 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.090318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v8d\" (UniqueName: \"kubernetes.io/projected/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-kube-api-access-d6v8d\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.119471 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.119423    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerStarted","Data":"46e383e26addea4fc9566f042bf9f8e95e05d495fa7408e93df367a4ac593630"}
Nov 03 10:28:20.121716 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.121694    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"2cb181debb656c62bb70f1078166094d","Type":"ContainerStarted","Data":"299b108faa17dcf3a53dedb8a4ce351eb7ba8cbad39ad2c039d631c363be1e0a"}
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.176908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.176954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwp9g\" (UniqueName: \"kubernetes.io/projected/316f1ae7-9bef-4691-a73d-e79f01fdea14-kube-api-access-rwp9g\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.176981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-system-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-sys\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-netns\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-hostroot\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-multus-certs\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565f923e-a384-496e-be31-f99ddaec54ce-tmp-dir\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0174417f-2cb0-4933-b9cc-7b044e2e8254-konnectivity-ca\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-lib-modules\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-socket-dir-parent\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-tmp\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/316f1ae7-9bef-4691-a73d-e79f01fdea14-iptables-alerter-script\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.183052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-conf-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-dbus\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-etc-kubernetes\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-kubelet-config\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-modprobe-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfqf\" (UniqueName: \"kubernetes.io/projected/cd31aaf2-4824-4357-818f-aaa8663c3917-kube-api-access-nbfqf\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-kubelet\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-daemon-config\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0174417f-2cb0-4933-b9cc-7b044e2e8254-agent-certs\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-kubernetes\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-run\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-cnibin\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-k8s-cni-cncf-io\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysconfig\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-systemd\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.183932 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-var-lib-kubelet\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/316f1ae7-9bef-4691-a73d-e79f01fdea14-host-slash\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/565f923e-a384-496e-be31-f99ddaec54ce-hosts-file\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstbl\" (UniqueName: \"kubernetes.io/projected/40cc057e-3d16-464d-8b2c-7a15d01238c1-kube-api-access-jstbl\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-cnibin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-host\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-tuned\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-cni-binary-copy\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.177982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4c6f\" (UniqueName: \"kubernetes.io/projected/565f923e-a384-496e-be31-f99ddaec54ce-kube-api-access-h4c6f\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-bin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-multus\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-conf\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmv4\" (UniqueName: \"kubernetes.io/projected/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-kube-api-access-cbmv4\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrz7t\" (UniqueName: \"kubernetes.io/projected/5e735188-d000-4d3e-9e65-f3320352f3b5-kube-api-access-jrz7t\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.184992 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.178980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.179207    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-system-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.179254    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-sys\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.179284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-netns\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.179307    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-hostroot\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.179337    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-multus-certs\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.179613    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565f923e-a384-496e-be31-f99ddaec54ce-tmp-dir\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.180105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0174417f-2cb0-4933-b9cc-7b044e2e8254-konnectivity-ca\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.180247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-lib-modules\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.180290    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.180742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.180880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.180992    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-socket-dir-parent\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.181390    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysconfig\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.181922    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/316f1ae7-9bef-4691-a73d-e79f01fdea14-iptables-alerter-script\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.181965    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-conf-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.182069    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-dbus\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.182109    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-etc-kubernetes\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.185885 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.182180    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-kubelet-config\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.182240    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.182299    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret podName:8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e nodeName:}" failed. No retries permitted until 2026-11-03 10:28:20.682279415 +0000 UTC m=+3.080239048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret") pod "global-pull-secret-syncer-zwj2x" (UID: "8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.182362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-modprobe-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.182563    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-kubelet\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183088    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-bin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/565f923e-a384-496e-be31-f99ddaec54ce-hosts-file\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183164    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183171    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-var-lib-kubelet\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/316f1ae7-9bef-4691-a73d-e79f01fdea14-host-slash\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183226    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-conf\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183229    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-run\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183249    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-cnibin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183280    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-systemd\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183298    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-multus\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183435    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-k8s-cni-cncf-io\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183459    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-kubernetes\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183483    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-cnibin\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.186746 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183531    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-host\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.183624    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183647    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-daemon-config\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.183684    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs podName:cd31aaf2-4824-4357-818f-aaa8663c3917 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:20.683667584 +0000 UTC m=+3.081627212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs") pod "network-metrics-daemon-4qhrr" (UID: "cd31aaf2-4824-4357-818f-aaa8663c3917") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183812    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-cni-binary-copy\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.183894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-tmp\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.185008    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.186198    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0174417f-2cb0-4933-b9cc-7b044e2e8254-agent-certs\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:20.187530 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.186428    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-tuned\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.190446 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.190420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwp9g\" (UniqueName: \"kubernetes.io/projected/316f1ae7-9bef-4691-a73d-e79f01fdea14-kube-api-access-rwp9g\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.191170 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.191143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrz7t\" (UniqueName: \"kubernetes.io/projected/5e735188-d000-4d3e-9e65-f3320352f3b5-kube-api-access-jrz7t\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.192205 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.192176    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfqf\" (UniqueName: \"kubernetes.io/projected/cd31aaf2-4824-4357-818f-aaa8663c3917-kube-api-access-nbfqf\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:20.193434 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.193416    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4c6f\" (UniqueName: \"kubernetes.io/projected/565f923e-a384-496e-be31-f99ddaec54ce-kube-api-access-h4c6f\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.193535 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.193495    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmv4\" (UniqueName: \"kubernetes.io/projected/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-kube-api-access-cbmv4\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.194220 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.194192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstbl\" (UniqueName: \"kubernetes.io/projected/40cc057e-3d16-464d-8b2c-7a15d01238c1-kube-api-access-jstbl\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.245868 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.245808    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:20.263292 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.263262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:28:20.275775 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:20.275745    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf53482_bc2a_4f95_a0ec_d2064baeb4f6.slice/crio-79e5efb821ab03a9dd347a84a09cf6686dadd40e44c445133bcc9d0217792f43 WatchSource:0}: Error finding container 79e5efb821ab03a9dd347a84a09cf6686dadd40e44c445133bcc9d0217792f43: Status 404 returned error can't find the container with id 79e5efb821ab03a9dd347a84a09cf6686dadd40e44c445133bcc9d0217792f43
Nov 03 10:28:20.280538 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.280515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:28:20.292938 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:20.292912    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ecc89_8209_4f18_bc24_e96628147236.slice/crio-0c101fb3f12fd969652b2400b34f924547f88df44aee188c0a5dd32ba347f492 WatchSource:0}: Error finding container 0c101fb3f12fd969652b2400b34f924547f88df44aee188c0a5dd32ba347f492: Status 404 returned error can't find the container with id 0c101fb3f12fd969652b2400b34f924547f88df44aee188c0a5dd32ba347f492
Nov 03 10:28:20.301758 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.301730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:28:20.304880 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.304865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:20.311972 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:20.311947    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1e2b76_dfe9_46d8_a82e_b837bbab1bfd.slice/crio-178e1f0aa9aea620e8717d019bd77f3dd96305af76de7ed2222af48754eaaec6 WatchSource:0}: Error finding container 178e1f0aa9aea620e8717d019bd77f3dd96305af76de7ed2222af48754eaaec6: Status 404 returned error can't find the container with id 178e1f0aa9aea620e8717d019bd77f3dd96305af76de7ed2222af48754eaaec6
Nov 03 10:28:20.315482 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:20.315453    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcd2d9d_803f_42cb_8db5_9f6a54c071c6.slice/crio-798e1f748fdb354fe892013fe551571574303c88599c06b8b117a3cbb6d91c0b WatchSource:0}: Error finding container 798e1f748fdb354fe892013fe551571574303c88599c06b8b117a3cbb6d91c0b: Status 404 returned error can't find the container with id 798e1f748fdb354fe892013fe551571574303c88599c06b8b117a3cbb6d91c0b
Nov 03 10:28:20.320324 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.320297    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:28:20.325392 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.325374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:28:20.327742 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.327720    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pf78w"
Nov 03 10:28:20.330701 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:20.330650    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316f1ae7_9bef_4691_a73d_e79f01fdea14.slice/crio-be6522254cd3b3401be94428e8dad0893e20b4e599345b87ccef0f1c48e337fd WatchSource:0}: Error finding container be6522254cd3b3401be94428e8dad0893e20b4e599345b87ccef0f1c48e337fd: Status 404 returned error can't find the container with id be6522254cd3b3401be94428e8dad0893e20b4e599345b87ccef0f1c48e337fd
Nov 03 10:28:20.331835 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.331813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:20.338875 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.338668    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:28:20.353201 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:28:20.353172    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565f923e_a384_496e_be31_f99ddaec54ce.slice/crio-2b9127068ef1d8b3ea9a2d4c74100d8b6ef16b6b1759a53bcab04ebb20a97bde WatchSource:0}: Error finding container 2b9127068ef1d8b3ea9a2d4c74100d8b6ef16b6b1759a53bcab04ebb20a97bde: Status 404 returned error can't find the container with id 2b9127068ef1d8b3ea9a2d4c74100d8b6ef16b6b1759a53bcab04ebb20a97bde
Nov 03 10:28:20.581796 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.581706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:20.581951 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.581919    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:20.582009 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.581949    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:20.582009 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.581966    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mlg5g for pod openshift-network-diagnostics/network-check-target-brkhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:20.582073 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.582033    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g podName:7075e9e0-fca9-44ba-82bf-e93c702b576d nodeName:}" failed. No retries permitted until 2026-11-03 10:28:21.582011681 +0000 UTC m=+3.979971296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlg5g" (UniqueName: "kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g") pod "network-check-target-brkhs" (UID: "7075e9e0-fca9-44ba-82bf-e93c702b576d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:20.683067 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.683028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:20.683265 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.683221    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:20.683326 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.683282    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret podName:8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e nodeName:}" failed. No retries permitted until 2026-11-03 10:28:21.683264514 +0000 UTC m=+4.081224150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret") pod "global-pull-secret-syncer-zwj2x" (UID: "8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:20.783581 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.783504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:20.783777 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.783665    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:20.783777 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:20.783741    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs podName:cd31aaf2-4824-4357-818f-aaa8663c3917 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:21.783721903 +0000 UTC m=+4.181681520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs") pod "network-metrics-daemon-4qhrr" (UID: "cd31aaf2-4824-4357-818f-aaa8663c3917") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:20.997560 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.997431    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:20 +0000 UTC" deadline="2028-08-07 02:50:53.165022522 +0000 UTC"
Nov 03 10:28:20.997560 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:20.997501    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15424h22m32.167526404s"
Nov 03 10:28:21.113625 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.112991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:21.113625 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.113131    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:21.131125 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.131084    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9bmpc" event={"ID":"565f923e-a384-496e-be31-f99ddaec54ce","Type":"ContainerStarted","Data":"2b9127068ef1d8b3ea9a2d4c74100d8b6ef16b6b1759a53bcab04ebb20a97bde"}
Nov 03 10:28:21.137676 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.137640    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pf78w" event={"ID":"5e735188-d000-4d3e-9e65-f3320352f3b5","Type":"ContainerStarted","Data":"7b48d395a1c66af09afa630d1d4c3b1173086b291478d651a40d154bc762f34c"}
Nov 03 10:28:21.156884 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.156807    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" event={"ID":"40cc057e-3d16-464d-8b2c-7a15d01238c1","Type":"ContainerStarted","Data":"97e54fd23d3668217337c906798e1e4c6183940adf814521684ba405c97b189f"}
Nov 03 10:28:21.162759 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.162707    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"798e1f748fdb354fe892013fe551571574303c88599c06b8b117a3cbb6d91c0b"}
Nov 03 10:28:21.164958 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.164931    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n" event={"ID":"b96ecc89-8209-4f18-bc24-e96628147236","Type":"ContainerStarted","Data":"0c101fb3f12fd969652b2400b34f924547f88df44aee188c0a5dd32ba347f492"}
Nov 03 10:28:21.166978 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.166945    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8v4lr" event={"ID":"0174417f-2cb0-4933-b9cc-7b044e2e8254","Type":"ContainerStarted","Data":"640c3fc7954e29e1655ba46d9513b818a199820e185329f86fc11fadc3a5ebfb"}
Nov 03 10:28:21.170363 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.170333    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6kr4v" event={"ID":"316f1ae7-9bef-4691-a73d-e79f01fdea14","Type":"ContainerStarted","Data":"be6522254cd3b3401be94428e8dad0893e20b4e599345b87ccef0f1c48e337fd"}
Nov 03 10:28:21.175419 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.175332    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fqldz" event={"ID":"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd","Type":"ContainerStarted","Data":"178e1f0aa9aea620e8717d019bd77f3dd96305af76de7ed2222af48754eaaec6"}
Nov 03 10:28:21.178838 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.178777    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxvr7" event={"ID":"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6","Type":"ContainerStarted","Data":"79e5efb821ab03a9dd347a84a09cf6686dadd40e44c445133bcc9d0217792f43"}
Nov 03 10:28:21.589834 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.589105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:21.589834 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.589273    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:21.589834 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.589295    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:21.589834 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.589320    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mlg5g for pod openshift-network-diagnostics/network-check-target-brkhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:21.589834 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.589370    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g podName:7075e9e0-fca9-44ba-82bf-e93c702b576d nodeName:}" failed. No retries permitted until 2026-11-03 10:28:23.589353326 +0000 UTC m=+5.987312937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlg5g" (UniqueName: "kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g") pod "network-check-target-brkhs" (UID: "7075e9e0-fca9-44ba-82bf-e93c702b576d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:21.690271 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.689912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:21.690271 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.690194    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:21.690486 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.690277    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret podName:8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e nodeName:}" failed. No retries permitted until 2026-11-03 10:28:23.690254888 +0000 UTC m=+6.088214730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret") pod "global-pull-secret-syncer-zwj2x" (UID: "8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:21.791635 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.791569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:21.792363 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.791731    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:21.792363 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:21.791949    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs podName:cd31aaf2-4824-4357-818f-aaa8663c3917 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:23.791928719 +0000 UTC m=+6.189888337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs") pod "network-metrics-daemon-4qhrr" (UID: "cd31aaf2-4824-4357-818f-aaa8663c3917") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:21.997954 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.997899    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:20 +0000 UTC" deadline="2028-04-07 01:42:38.070009004 +0000 UTC"
Nov 03 10:28:21.997954 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:21.997931    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12495h14m16.072081494s"
Nov 03 10:28:22.120676 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:22.118254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:22.120676 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:22.118409    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:22.120676 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:22.120408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:22.120676 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:22.120579    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:22.192171 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:22.191686    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"2cb181debb656c62bb70f1078166094d","Type":"ContainerStarted","Data":"776bcc34a786d8793385bfb78b24fe165a1bbcf1918f3ef2300bbcce41e05eb1"}
Nov 03 10:28:22.199688 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:22.199217    2576 generic.go:356] "Generic (PLEG): container finished" podID="b0906d8be50774c4e8328161c8119d16" containerID="1204ee23ba3264cda244c0c385c03626b81ebb4306b3c411e559a8813163890e" exitCode=0
Nov 03 10:28:22.199688 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:22.199263    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerDied","Data":"1204ee23ba3264cda244c0c385c03626b81ebb4306b3c411e559a8813163890e"}
Nov 03 10:28:22.219455 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:22.211216    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" podStartSLOduration=3.21119733 podStartE2EDuration="3.21119733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:22.209588004 +0000 UTC m=+4.607547646" watchObservedRunningTime="2026-11-03 10:28:22.21119733 +0000 UTC m=+4.609156969"
Nov 03 10:28:23.113554 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:23.113527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:23.114023 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.113679    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:23.221586 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:23.221550    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerStarted","Data":"5b11a1ef1302dbe6f933adcbc3efb545011d6add15a32b33cd160c821e136c79"}
Nov 03 10:28:23.237512 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:23.236638    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" podStartSLOduration=4.236618826 podStartE2EDuration="4.236618826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:23.236517871 +0000 UTC m=+5.634477508" watchObservedRunningTime="2026-11-03 10:28:23.236618826 +0000 UTC m=+5.634578463"
Nov 03 10:28:23.608233 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:23.607980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:23.608233 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.608151    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:23.608233 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.608176    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:23.608233 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.608190    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mlg5g for pod openshift-network-diagnostics/network-check-target-brkhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:23.608586 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.608253    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g podName:7075e9e0-fca9-44ba-82bf-e93c702b576d nodeName:}" failed. No retries permitted until 2026-11-03 10:28:27.608233994 +0000 UTC m=+10.006193613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlg5g" (UniqueName: "kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g") pod "network-check-target-brkhs" (UID: "7075e9e0-fca9-44ba-82bf-e93c702b576d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:23.709349 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:23.709263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:23.709547 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.709527    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:23.709624 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.709605    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret podName:8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e nodeName:}" failed. No retries permitted until 2026-11-03 10:28:27.709584782 +0000 UTC m=+10.107544397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret") pod "global-pull-secret-syncer-zwj2x" (UID: "8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:23.810550 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:23.809823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:23.810550 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.810005    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:23.810550 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:23.810089    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs podName:cd31aaf2-4824-4357-818f-aaa8663c3917 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:27.810068806 +0000 UTC m=+10.208028422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs") pod "network-metrics-daemon-4qhrr" (UID: "cd31aaf2-4824-4357-818f-aaa8663c3917") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:24.117781 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:24.116829    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:24.117781 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:24.116963    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:24.117781 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:24.117376    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:24.117781 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:24.117509    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:24.232002 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:24.231963    2576 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="f8a2105165e83c189d7fcec4f77ee68376b9189f49170e2d77e54ce1cb4ec527" exitCode=0
Nov 03 10:28:24.232200 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:24.232100    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" event={"ID":"40cc057e-3d16-464d-8b2c-7a15d01238c1","Type":"ContainerDied","Data":"f8a2105165e83c189d7fcec4f77ee68376b9189f49170e2d77e54ce1cb4ec527"}
Nov 03 10:28:25.113892 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:25.113859    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:25.114096 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:25.113994    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:25.250202 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:25.249439    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8v4lr" event={"ID":"0174417f-2cb0-4933-b9cc-7b044e2e8254","Type":"ContainerStarted","Data":"74685632e59923cfaff24bb5d96c74cbec9958905786fba85ff6cc621b87773e"}
Nov 03 10:28:26.114795 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:26.114713    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:26.114964 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:26.114880    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:26.115033 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:26.114978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:26.115090 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:26.115069    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:26.255597 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:26.255552    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxvr7" event={"ID":"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6","Type":"ContainerStarted","Data":"7bd18964f0492d4de8ee48f22f611f6254bbbdd0fb2d8ae88522fdcb28466a30"}
Nov 03 10:28:26.270255 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:26.270188    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8v4lr" podStartSLOduration=3.2177704289999998 podStartE2EDuration="7.270168343s" totalImagesPullingTime="4.052397914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:25.266873402 +0000 UTC m=+7.664833039" watchObservedRunningTime="2026-11-03 10:28:26.270168343 +0000 UTC m=+8.668127980"
Nov 03 10:28:27.113328 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:27.113282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:27.113522 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.113419    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:27.261897 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:27.261836    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n" event={"ID":"b96ecc89-8209-4f18-bc24-e96628147236","Type":"ContainerStarted","Data":"be5cac779d11694e245f11d62869c111c82f6277a61590dbc83ee0846dc35a2a"}
Nov 03 10:28:27.643234 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:27.642580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:27.643234 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.642754    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:27.643234 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.642777    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:27.643234 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.642790    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mlg5g for pod openshift-network-diagnostics/network-check-target-brkhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:27.643234 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.642844    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g podName:7075e9e0-fca9-44ba-82bf-e93c702b576d nodeName:}" failed. No retries permitted until 2026-11-03 10:28:35.64282631 +0000 UTC m=+18.040785927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlg5g" (UniqueName: "kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g") pod "network-check-target-brkhs" (UID: "7075e9e0-fca9-44ba-82bf-e93c702b576d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:27.743303 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:27.743271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:27.743491 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.743463    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:27.743560 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.743526    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret podName:8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e nodeName:}" failed. No retries permitted until 2026-11-03 10:28:35.743507574 +0000 UTC m=+18.141467206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret") pod "global-pull-secret-syncer-zwj2x" (UID: "8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:27.844670 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:27.844629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:27.844856 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.844774    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:27.844856 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:27.844839    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs podName:cd31aaf2-4824-4357-818f-aaa8663c3917 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:35.844821266 +0000 UTC m=+18.242780883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs") pod "network-metrics-daemon-4qhrr" (UID: "cd31aaf2-4824-4357-818f-aaa8663c3917") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:28.114235 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:28.114167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:28.114359 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:28.114290    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:28.114757 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:28.114724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:28.114890 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:28.114844    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:28.322345 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:28.322272    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 10:28:28.887765 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:28.887656    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:28.888452 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:28.888389    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:28.904372 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:28.904323    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cxvr7" podStartSLOduration=5.514753142 podStartE2EDuration="10.904307082s" totalImagesPullingTime="5.38955394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:26.27091019 +0000 UTC m=+8.668869826" watchObservedRunningTime="2026-11-03 10:28:28.904307082 +0000 UTC m=+11.302266721"
Nov 03 10:28:29.028310 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.028203    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T10:28:28.322294983Z","UUID":"e5ddc60b-da8d-48bd-963d-edc051e4ee50","Handler":null,"Name":"","Endpoint":""}
Nov 03 10:28:29.032964 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.032943    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 10:28:29.033099 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.032971    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 10:28:29.113314 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.113281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:29.113506 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:29.113427    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:29.274856 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.274174    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9bmpc" event={"ID":"565f923e-a384-496e-be31-f99ddaec54ce","Type":"ContainerStarted","Data":"49ba9293608554004b285af10747891d5af3032da0f906d99de235e3775aa6b5"}
Nov 03 10:28:29.276237 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.276165    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pf78w" event={"ID":"5e735188-d000-4d3e-9e65-f3320352f3b5","Type":"ContainerStarted","Data":"4ee8ff86829f21d7e4f363f5fb3b22405ff9f586192c0f000230b507e9fed936"}
Nov 03 10:28:29.279041 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.279011    2576 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="23f8f0b8400e95c71f455e421077b7af835de3c653122302b157853df719169e" exitCode=0
Nov 03 10:28:29.279155 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.279098    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" event={"ID":"40cc057e-3d16-464d-8b2c-7a15d01238c1","Type":"ContainerDied","Data":"23f8f0b8400e95c71f455e421077b7af835de3c653122302b157853df719169e"}
Nov 03 10:28:29.281718 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.281696    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n" event={"ID":"b96ecc89-8209-4f18-bc24-e96628147236","Type":"ContainerStarted","Data":"9f9dd125855ab1002baa1e9c7f180d0f332ec0f859c75889f75b6c853b3bec92"}
Nov 03 10:28:29.281879 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.281858    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:29.282422 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.282404    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:28:29.289343 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.289291    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9bmpc" podStartSLOduration=1.93375544 podStartE2EDuration="10.289276504s" totalImagesPullingTime="8.355521064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:29.289112772 +0000 UTC m=+11.687072409" watchObservedRunningTime="2026-11-03 10:28:29.289276504 +0000 UTC m=+11.687236141"
Nov 03 10:28:29.344680 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.344603    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pf78w" podStartSLOduration=2.51891722 podStartE2EDuration="10.344586094s" totalImagesPullingTime="7.825668874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:29.32929925 +0000 UTC m=+11.727258894" watchObservedRunningTime="2026-11-03 10:28:29.344586094 +0000 UTC m=+11.742545752"
Nov 03 10:28:29.523092 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.522896    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:29.717638 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.717495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:29.922829 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:29.922777    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:30.114068 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.114043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:30.114262 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:30.114232    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:30.114330 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.114280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:30.114420 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:30.114398    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:30.285603 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.285519    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6kr4v" event={"ID":"316f1ae7-9bef-4691-a73d-e79f01fdea14","Type":"ContainerStarted","Data":"0bde0875b548e05e47e41d6900d25dd3737a35b8ca952e2b3eb19201ca13b9b7"}
Nov 03 10:28:30.287836 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.287808    2576 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="3b365a1cccbe5c491b94ede9dafac81819aad4fdb62f9a5497629c99f86883e1" exitCode=0
Nov 03 10:28:30.287933 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.287896    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" event={"ID":"40cc057e-3d16-464d-8b2c-7a15d01238c1","Type":"ContainerDied","Data":"3b365a1cccbe5c491b94ede9dafac81819aad4fdb62f9a5497629c99f86883e1"}
Nov 03 10:28:30.290215 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.290194    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n" event={"ID":"b96ecc89-8209-4f18-bc24-e96628147236","Type":"ContainerStarted","Data":"7ebc98f5d17aec089b6fd3983dbb9bccbb35917d185be9b14514a11a2671ed3f"}
Nov 03 10:28:30.306903 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.306852    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6kr4v" podStartSLOduration=3.011732504 podStartE2EDuration="11.306836102s" totalImagesPullingTime="8.295103598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:30.306260414 +0000 UTC m=+12.704220112" watchObservedRunningTime="2026-11-03 10:28:30.306836102 +0000 UTC m=+12.704795741"
Nov 03 10:28:30.325779 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.325728    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n" podStartSLOduration=3.481765303 podStartE2EDuration="12.325717002s" totalImagesPullingTime="8.843951699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:30.324983334 +0000 UTC m=+12.722942973" watchObservedRunningTime="2026-11-03 10:28:30.325717002 +0000 UTC m=+12.723676639"
Nov 03 10:28:30.480483 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.480454    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:30.608115 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:30.608031    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:31.113257 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:31.113238    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:31.113379 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:31.113357    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:31.295514 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:31.295439    2576 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="d8665006caa700bc799fd4b1b563539b41150751f29fc014d84a458d8de3571e" exitCode=0
Nov 03 10:28:31.295681 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:31.295651    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" event={"ID":"40cc057e-3d16-464d-8b2c-7a15d01238c1","Type":"ContainerDied","Data":"d8665006caa700bc799fd4b1b563539b41150751f29fc014d84a458d8de3571e"}
Nov 03 10:28:32.113325 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:32.113293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:32.113841 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:32.113436    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:32.113841 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:32.113487    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:32.113841 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:32.113637    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:33.113715 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:33.113685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:33.114341 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:33.113805    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:34.102265 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:34.102232    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:34.113209 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:34.113183    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:34.113209 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:34.113208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:34.113387 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:34.113313    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:34.113480 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:34.113454    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:34.233897 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:34.233453    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:34.305742 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:34.305532    2576 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="c2cffd239a94fe5775cfc60026149e5932cbb85769e5af97a1f063711065c59d" exitCode=0
Nov 03 10:28:34.305939 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:34.305611    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" event={"ID":"40cc057e-3d16-464d-8b2c-7a15d01238c1","Type":"ContainerDied","Data":"c2cffd239a94fe5775cfc60026149e5932cbb85769e5af97a1f063711065c59d"}
Nov 03 10:28:34.307356 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:34.307307    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fqldz" event={"ID":"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd","Type":"ContainerStarted","Data":"209b1f6427cf18c434ad3d99a03be12ad50373ceb3412ee0d8de4e1c1c001241"}
Nov 03 10:28:34.342309 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:34.342230    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fqldz" podStartSLOduration=2.525172692 podStartE2EDuration="16.342211643s" totalImagesPullingTime="13.817038951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:34.341834218 +0000 UTC m=+16.739793852" watchObservedRunningTime="2026-11-03 10:28:34.342211643 +0000 UTC m=+16.740171276"
Nov 03 10:28:35.113270 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.113244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:35.113463 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.113344    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:35.235198 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.235169    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:35.311689 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.311664    2576 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="699d5f6b5dd51b089e1855624c49fe6a2c41ec0c23db679baee1b0e5f4b86595" exitCode=0
Nov 03 10:28:35.311827 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.311748    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" event={"ID":"40cc057e-3d16-464d-8b2c-7a15d01238c1","Type":"ContainerDied","Data":"699d5f6b5dd51b089e1855624c49fe6a2c41ec0c23db679baee1b0e5f4b86595"}
Nov 03 10:28:35.376732 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.376713    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:35.546619 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.546590    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:35.682586 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.682502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:35.709455 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.709428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:35.709567 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.709535    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:35.709567 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.709555    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:35.709567 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.709563    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mlg5g for pod openshift-network-diagnostics/network-check-target-brkhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:35.709702 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.709623    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g podName:7075e9e0-fca9-44ba-82bf-e93c702b576d nodeName:}" failed. No retries permitted until 2026-11-03 10:28:51.709610602 +0000 UTC m=+34.107570215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlg5g" (UniqueName: "kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g") pod "network-check-target-brkhs" (UID: "7075e9e0-fca9-44ba-82bf-e93c702b576d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:35.809731 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.809709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:35.809846 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.809831    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:35.809914 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.809893    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret podName:8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e nodeName:}" failed. No retries permitted until 2026-11-03 10:28:51.809874878 +0000 UTC m=+34.207834503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret") pod "global-pull-secret-syncer-zwj2x" (UID: "8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:35.814853 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.814835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:35.910214 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:35.910197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:35.910306 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.910291    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:35.910365 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:35.910332    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs podName:cd31aaf2-4824-4357-818f-aaa8663c3917 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:51.910318111 +0000 UTC m=+34.308277726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs") pod "network-metrics-daemon-4qhrr" (UID: "cd31aaf2-4824-4357-818f-aaa8663c3917") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:36.113854 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.113825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:36.114004 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:36.113980    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:36.114081 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.114044    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:36.114188 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:36.114166    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:36.316430 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.316347    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" event={"ID":"40cc057e-3d16-464d-8b2c-7a15d01238c1","Type":"ContainerStarted","Data":"cb05aae8d2ec4473a5678bece18a4ed828ed51ef529e9912425aba64a3e67c3b"}
Nov 03 10:28:36.322320 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.322292    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"e444e10570c697a9b8b9865b2a284c4cebd773684ea0c6b3b0c89f13278a72ff"}
Nov 03 10:28:36.322445 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.322379    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"33096517287a0e7ee7ad048cfada9cbfa84ffe45b1a73f61e9f9b7c94166cd32"}
Nov 03 10:28:36.322445 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.322416    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"0d957a034735c6413b5c945c400498b5edadf359989f6046cb1fad20f1b070f2"}
Nov 03 10:28:36.322509 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.322447    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"d3b1e8a57c1cdb9afbbe3ccc9b682e0ecebe717c492ddce8e4b408a889873f0a"}
Nov 03 10:28:36.322509 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.322474    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"0a2b668d3d0cfb76fbd3ae80d85a038ce65c1c2227f9e3ea0a9cdfa53897388d"}
Nov 03 10:28:36.322564 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.322508    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"ff071aefb4f3e63ad1fe509b7655732f7b9643348bbc8a3680e73111258783d5"}
Nov 03 10:28:36.339009 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.338963    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jtzkm" podStartSLOduration=6.854241911 podStartE2EDuration="18.338952046s" totalImagesPullingTime="11.484710135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:36.338556069 +0000 UTC m=+18.736515702" watchObservedRunningTime="2026-11-03 10:28:36.338952046 +0000 UTC m=+18.736911679"
Nov 03 10:28:36.439648 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.439621    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:36.590733 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.590640    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:36.727690 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.727515    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:36.854353 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.854264    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:36.984400 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:36.984374    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:37.111466 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:37.111399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:37.113415 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:37.113401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:37.113499 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:37.113483    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:37.959369 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:37.959338    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:38.088417 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:38.088398    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:38.114241 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:38.114217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:38.114313 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:38.114306    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:38.114376 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:38.114352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:38.114416 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:38.114362    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:38.328789 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:38.328718    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"7523e87ed96b1f59ba1212b2cc25412290328287f196c4c276c06f2612836a9b"}
Nov 03 10:28:38.448447 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:38.448414    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:38.582368 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:38.582307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:38.868052 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:38.867971    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:38.994335 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:38.994308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:39.049693 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:39.049668    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9bmpc_565f923e-a384-496e-be31-f99ddaec54ce/dns-node-resolver/0.log"
Nov 03 10:28:39.113043 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:39.113020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:39.113148 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:39.113129    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:39.121752 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:39.121703    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:39.759142 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:39.759111    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:40.113446 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.113379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:40.113446 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.113408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:40.113975 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:40.113502    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:40.113975 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:40.113626    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:40.168123 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.168100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:40.244611 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.244594    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxvr7_3bf53482-bc2a-4f95-a0ec-d2064baeb4f6/node-ca/0.log"
Nov 03 10:28:40.337259 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.337231    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" event={"ID":"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6","Type":"ContainerStarted","Data":"5a77d89c457cece4bd8a37d310f76c57867e30db8ddd5c5e11bb7fb219574fbf"}
Nov 03 10:28:40.337416 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.337398    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:40.337472 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.337456    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:40.337520 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.337488    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:40.355006 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.354981    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:40.355171 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.355157    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:28:40.364431 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.364354    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm" podStartSLOduration=6.92695743 podStartE2EDuration="22.364343851s" totalImagesPullingTime="15.437386421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:40.363942516 +0000 UTC m=+22.761902150" watchObservedRunningTime="2026-11-03 10:28:40.364343851 +0000 UTC m=+22.762303482"
Nov 03 10:28:40.370795 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.370773    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:40.566347 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.566317    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:40.940875 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:40.940844    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:41.073798 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:41.073762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:41.112852 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:41.112828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:41.112947 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:41.112922    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:41.208001 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:41.207935    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:41.336909 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:41.336884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:41.758934 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:41.758900    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:41.890008 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:41.889983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:42.028673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:42.028615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:42.113215 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:42.113193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:42.113361 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:42.113339    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:42.113413 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:42.113378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:42.113455 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:42.113442    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:42.265240 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:42.265211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:42.404070 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:42.404009    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:42.560831 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:42.560807    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:42.960996 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:42.960969    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:43.113043 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:43.113014    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:43.113216 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:43.113120    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:43.182647 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:43.182621    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:43.330766 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:43.330696    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:43.772359 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:43.772274    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:44.113808 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:44.113737    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:44.113963 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:44.113745    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:44.113963 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:44.113859    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:44.113963 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:44.113931    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:44.272220 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:44.272192    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:44.428899 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:44.428869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:44.557806 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:44.557781    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:44.687647 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:44.687586    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:44.821528 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:44.821500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:44.970763 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:44.970696    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:45.101148 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:45.101121    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:45.113185 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:45.113164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:45.113292 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:45.113273    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:45.231433 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:45.231373    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:45.368730 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:45.368706    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:45.758394 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:45.758361    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:46.115630 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:46.115558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:46.115756 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:46.115720    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:46.115805 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:46.115767    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:46.115855 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:46.115835    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:46.189765 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:46.189736    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:46.558418 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:46.558391    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:46.958722 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:46.958699    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:47.113651 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:47.113630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:47.113779 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:47.113719    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:47.159254 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:47.159227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:47.335508 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:47.335447    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:47.760880 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:47.760851    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:48.113769 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:48.113689    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:48.113769 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:48.113764    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:48.114258 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:48.113773    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:48.114258 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:48.113856    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:48.364224 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:48.364132    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:48.563054 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:48.563030    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:48.958224 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:48.958195    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:49.113755 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:49.113733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:49.113895 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:49.113821    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:49.164578 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:49.164553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:49.564154 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:49.564116    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:49.701459 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:49.701431    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:49.826648 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:49.826562    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:50.113393 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:50.113318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:50.113543 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:50.113455    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:50.113543 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:50.113490    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:50.113638 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:50.113551    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:50.164498 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:50.164470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:51.113764 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:51.113730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:51.113921 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.113831    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:51.725841 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:51.725813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:51.726303 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.725929    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:51.726303 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.725946    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:51.726303 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.725955    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mlg5g for pod openshift-network-diagnostics/network-check-target-brkhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:51.726303 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.725997    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g podName:7075e9e0-fca9-44ba-82bf-e93c702b576d nodeName:}" failed. No retries permitted until 2026-11-03 10:29:23.725985749 +0000 UTC m=+66.123945358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlg5g" (UniqueName: "kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g") pod "network-check-target-brkhs" (UID: "7075e9e0-fca9-44ba-82bf-e93c702b576d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:51.826062 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:51.826036    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:51.826187 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.826122    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:51.826187 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.826169    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret podName:8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e nodeName:}" failed. No retries permitted until 2026-11-03 10:29:23.826154358 +0000 UTC m=+66.224113969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret") pod "global-pull-secret-syncer-zwj2x" (UID: "8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:51.926436 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:51.926407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:51.926561 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.926488    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:51.926561 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:51.926530    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs podName:cd31aaf2-4824-4357-818f-aaa8663c3917 nodeName:}" failed. No retries permitted until 2026-11-03 10:29:23.926518785 +0000 UTC m=+66.324478395 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs") pod "network-metrics-daemon-4qhrr" (UID: "cd31aaf2-4824-4357-818f-aaa8663c3917") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:52.113851 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:52.113774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:52.113980 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:52.113882    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:52.113980 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:52.113925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:52.113980 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:52.113976    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:53.113733 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:53.113700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:53.114122 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:53.113804    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:54.113485 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:54.113456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:54.113485 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:54.113486    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:54.113693 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:54.113557    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:54.113693 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:54.113677    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:55.113725 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:55.113694    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:55.114091 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:55.113781    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:56.113758 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:56.113726    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:56.114144 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:56.113856    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:56.114144 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:56.113905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:56.114144 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:56.113997    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:56.741651 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:56.741616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:56.869863 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:56.869827    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:56.990400 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:56.990376    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:57.113783 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:57.113731    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:57.114098 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:57.113812    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:57.124276 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:57.124252    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:57.830349 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:57.830323    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:58.027382 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:58.027353    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:58.115044 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:58.114818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:58.115386 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:58.115048    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:58.115386 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:58.114888    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:58.115386 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:58.115131    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:28:58.165499 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:58.165475    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:58.298128 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:58.298103    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:28:59.041603 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:59.041493    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zwj2x"]
Nov 03 10:28:59.041804 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:59.041641    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:28:59.041804 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:59.041748    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:28:59.044682 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:59.044660    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-brkhs"]
Nov 03 10:28:59.044913 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:59.044896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:28:59.045018 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:59.045000    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:28:59.045362 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:59.045336    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4qhrr"]
Nov 03 10:28:59.045451 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:28:59.045429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:28:59.045523 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:28:59.045507    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:29:00.843719 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:00.843688    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:00.976512 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:00.976463    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:01.100750 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:01.100661    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:01.113665 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:01.113635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:01.113665 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:01.113670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:01.113844 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:01.113689    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:01.113844 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:29:01.113789    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:29:01.113937 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:29:01.113868    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:29:01.113937 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:29:01.113911    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:29:03.113200 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.113171    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:03.113200 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.113191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:03.113614 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:29:03.113285    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qhrr" podUID="cd31aaf2-4824-4357-818f-aaa8663c3917"
Nov 03 10:29:03.113614 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.113340    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:03.113614 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:29:03.113422    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brkhs" podUID="7075e9e0-fca9-44ba-82bf-e93c702b576d"
Nov 03 10:29:03.113614 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:29:03.113525    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwj2x" podUID="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e"
Nov 03 10:29:03.590473 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.590445    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:03.881216 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.881141    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeReady"
Nov 03 10:29:03.881372 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.881266    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 10:29:03.920153 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.920124    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jq8jl"]
Nov 03 10:29:03.921943 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.921928    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:03.923063 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.923042    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d9x57"]
Nov 03 10:29:03.924542 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.924527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:03.926127 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.926106    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5cc62\""
Nov 03 10:29:03.926233 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.926111    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 10:29:03.926233 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.926114    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 10:29:03.926233 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.926146    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 10:29:03.926233 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.926151    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 10:29:03.935530 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.935510    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jq8jl"]
Nov 03 10:29:03.936199 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.936150    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d9x57"]
Nov 03 10:29:03.941296 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.941278    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wtrwm"]
Nov 03 10:29:03.943034 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.943018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:03.954502 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:03.954483    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wtrwm"]
Nov 03 10:29:04.026321 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.026293    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 10:29:04.026431 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.026326    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 10:29:04.026431 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.026361    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 10:29:04.026521 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.026507    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w9nrb\""
Nov 03 10:29:04.026565 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.026505    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 10:29:04.026565 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.026556    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 10:29:04.026669 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.026639    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w9j8\""
Nov 03 10:29:04.112964 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.112936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254ba5e2-5f70-4980-9f35-705260e6b4a8-cert\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:04.113072 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.112971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcnv\" (UniqueName: \"kubernetes.io/projected/d613c8d4-0ecb-4178-839a-e626bd2646e5-kube-api-access-6vcnv\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.113072 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.112997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d613c8d4-0ecb-4178-839a-e626bd2646e5-tmp-dir\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.113072 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.113020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.113205 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.113082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/487f4942-ea5d-44a2-92dc-9fed886c9fdf-data-volume\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.113239 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.113197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d613c8d4-0ecb-4178-839a-e626bd2646e5-config-volume\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.113545 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.113254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/487f4942-ea5d-44a2-92dc-9fed886c9fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.113545 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.113282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cbk\" (UniqueName: \"kubernetes.io/projected/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-api-access-g2cbk\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.113545 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.113311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw56c\" (UniqueName: \"kubernetes.io/projected/254ba5e2-5f70-4980-9f35-705260e6b4a8-kube-api-access-fw56c\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:04.113545 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.113365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d613c8d4-0ecb-4178-839a-e626bd2646e5-metrics-tls\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.113545 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.113394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/487f4942-ea5d-44a2-92dc-9fed886c9fdf-crio-socket\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.213795 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.213765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/487f4942-ea5d-44a2-92dc-9fed886c9fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.214006 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.213800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cbk\" (UniqueName: \"kubernetes.io/projected/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-api-access-g2cbk\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.214006 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.213817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw56c\" (UniqueName: \"kubernetes.io/projected/254ba5e2-5f70-4980-9f35-705260e6b4a8-kube-api-access-fw56c\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:04.214006 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.213838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d613c8d4-0ecb-4178-839a-e626bd2646e5-metrics-tls\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.214006 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.213853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/487f4942-ea5d-44a2-92dc-9fed886c9fdf-crio-socket\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.214006 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.213988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254ba5e2-5f70-4980-9f35-705260e6b4a8-cert\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:04.214263 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcnv\" (UniqueName: \"kubernetes.io/projected/d613c8d4-0ecb-4178-839a-e626bd2646e5-kube-api-access-6vcnv\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.214263 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d613c8d4-0ecb-4178-839a-e626bd2646e5-tmp-dir\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.214263 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214086    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/487f4942-ea5d-44a2-92dc-9fed886c9fdf-crio-socket\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.214263 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.214263 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/487f4942-ea5d-44a2-92dc-9fed886c9fdf-data-volume\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.214263 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d613c8d4-0ecb-4178-839a-e626bd2646e5-config-volume\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.214539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d613c8d4-0ecb-4178-839a-e626bd2646e5-tmp-dir\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.214622 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214608    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/487f4942-ea5d-44a2-92dc-9fed886c9fdf-data-volume\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.214801 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214781    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d613c8d4-0ecb-4178-839a-e626bd2646e5-config-volume\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.214852 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.214799    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.218045 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.217946    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d613c8d4-0ecb-4178-839a-e626bd2646e5-metrics-tls\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.218141 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.218017    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/487f4942-ea5d-44a2-92dc-9fed886c9fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.218141 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.218095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254ba5e2-5f70-4980-9f35-705260e6b4a8-cert\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:04.221860 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.221841    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcnv\" (UniqueName: \"kubernetes.io/projected/d613c8d4-0ecb-4178-839a-e626bd2646e5-kube-api-access-6vcnv\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.221949 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.221895    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cbk\" (UniqueName: \"kubernetes.io/projected/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-api-access-g2cbk\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.222173 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.222154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw56c\" (UniqueName: \"kubernetes.io/projected/254ba5e2-5f70-4980-9f35-705260e6b4a8-kube-api-access-fw56c\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:04.235861 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.235845    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:04.239100 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.239085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:04.255712 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.255688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:04.290778 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.290746    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:04.451712 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.451679    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:04.530845 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.530775    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d9x57"]
Nov 03 10:29:04.534641 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.534618    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wtrwm"]
Nov 03 10:29:04.534763 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.534675    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jq8jl"]
Nov 03 10:29:04.535051 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:04.535029    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254ba5e2_5f70_4980_9f35_705260e6b4a8.slice/crio-3e47fab9093886b012e2a6436c400d52895195cf1906a2c544d8545c05cb5b91 WatchSource:0}: Error finding container 3e47fab9093886b012e2a6436c400d52895195cf1906a2c544d8545c05cb5b91: Status 404 returned error can't find the container with id 3e47fab9093886b012e2a6436c400d52895195cf1906a2c544d8545c05cb5b91
Nov 03 10:29:04.537275 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:04.537249    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487f4942_ea5d_44a2_92dc_9fed886c9fdf.slice/crio-d4fe728f134e0fdee62072b73b006a18cc2a5f5288b6f86df64dbd3dcdc765e7 WatchSource:0}: Error finding container d4fe728f134e0fdee62072b73b006a18cc2a5f5288b6f86df64dbd3dcdc765e7: Status 404 returned error can't find the container with id d4fe728f134e0fdee62072b73b006a18cc2a5f5288b6f86df64dbd3dcdc765e7
Nov 03 10:29:04.537895 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:04.537873    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd613c8d4_0ecb_4178_839a_e626bd2646e5.slice/crio-f03d2523fff92b2dc272ef48c88c1a615f4990cbdab28aa4a8f34e39833cc96e WatchSource:0}: Error finding container f03d2523fff92b2dc272ef48c88c1a615f4990cbdab28aa4a8f34e39833cc96e: Status 404 returned error can't find the container with id f03d2523fff92b2dc272ef48c88c1a615f4990cbdab28aa4a8f34e39833cc96e
Nov 03 10:29:04.591346 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:04.591320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:05.008016 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.007968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:05.113507 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.113477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:05.113507 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.113501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:05.114314 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.113732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:05.117177 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.117154    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 10:29:05.118480 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.118456    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 10:29:05.118664 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.118583    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 10:29:05.118664 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.118642    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pthxs\""
Nov 03 10:29:05.118795 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.118669    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 10:29:05.118795 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.118677    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ssvsh\""
Nov 03 10:29:05.118795 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.118729    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lq8vs\""
Nov 03 10:29:05.163407 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.163365    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"]
Nov 03 10:29:05.165340 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.165318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.171998 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.171966    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"]
Nov 03 10:29:05.174084 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.174065    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.180354 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.180328    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"]
Nov 03 10:29:05.183198 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.183065    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:05.188631 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.188608    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"]
Nov 03 10:29:05.192539 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.192518    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mpkfs"]
Nov 03 10:29:05.194669 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.194650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.209230 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.209209    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 10:29:05.209501 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.209471    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 10:29:05.209501 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.209478    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 10:29:05.209768 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.209592    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gzgf5\""
Nov 03 10:29:05.211870 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.211853    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 10:29:05.212025 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.211869    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 10:29:05.212094 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.211908    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-szl4w\""
Nov 03 10:29:05.212094 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.211928    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjjb4\""
Nov 03 10:29:05.212288 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.212274    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 10:29:05.221397 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xzg\" (UniqueName: \"kubernetes.io/projected/6aa3275d-c0c8-4ffb-9264-280b10d1672c-kube-api-access-t9xzg\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.221505 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.221505 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddv27\" (UniqueName: \"kubernetes.io/projected/ac2fcf63-43d0-4759-9e77-edd3500ff388-kube-api-access-ddv27\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.221505 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-textfile\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.221672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-tls\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.221672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqfgx\" (UniqueName: \"kubernetes.io/projected/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-api-access-nqfgx\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.221672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221602    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.221672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.221672 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac2fcf63-43d0-4759-9e77-edd3500ff388-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.221828 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.221871 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.221871 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.221947 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-root\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.221947 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-wtmp\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.222033 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.221981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.222033 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.222006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.222109 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.222034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.222150 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.222123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-metrics-client-ca\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.222186 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.222158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-sys\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.224702 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.224515    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 10:29:05.224702 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.224532    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 10:29:05.225012 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.224967    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 10:29:05.225107 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.225081    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 10:29:05.225182 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.225158    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 10:29:05.245813 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.245794    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9bmpc_565f923e-a384-496e-be31-f99ddaec54ce/dns-node-resolver/0.log"
Nov 03 10:29:05.322668 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.322782 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.322782 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.322782 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-root\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.322782 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-wtmp\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.322988 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.322988 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.322988 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.322988 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-metrics-client-ca\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.322988 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-sys\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.322988 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xzg\" (UniqueName: \"kubernetes.io/projected/6aa3275d-c0c8-4ffb-9264-280b10d1672c-kube-api-access-t9xzg\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.323360 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.322991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.323360 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.323018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddv27\" (UniqueName: \"kubernetes.io/projected/ac2fcf63-43d0-4759-9e77-edd3500ff388-kube-api-access-ddv27\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.323360 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.323048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-textfile\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.323360 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.323077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-tls\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.323360 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.323118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqfgx\" (UniqueName: \"kubernetes.io/projected/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-api-access-nqfgx\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.323360 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.323150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.323360 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.323183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.323360 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.323208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac2fcf63-43d0-4759-9e77-edd3500ff388-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.323946 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:29:05.323913    2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Nov 03 10:29:05.324047 ip-10-0-138-233 kubenswrapper[2576]: E1103 10:29:05.323994    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls podName:ac2fcf63-43d0-4759-9e77-edd3500ff388 nodeName:}" failed. No retries permitted until 2026-11-03 10:29:05.823975007 +0000 UTC m=+48.221934621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls") pod "openshift-state-metrics-7bd7489957-8lfdz" (UID: "ac2fcf63-43d0-4759-9e77-edd3500ff388") : secret "openshift-state-metrics-tls" not found
Nov 03 10:29:05.324047 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.323998    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac2fcf63-43d0-4759-9e77-edd3500ff388-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.324047 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.324018    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-metrics-client-ca\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.324221 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.324057    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-textfile\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.324221 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.324121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-sys\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.324326 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.324232    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-wtmp\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.324380 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.324327    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.324431 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.324347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-root\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.325816 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.325071    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.325816 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.325189    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.325816 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.325772    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.327648 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.327612    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.327648 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.327621    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.327802 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.327730    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.328056 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.328038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.329818 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.329794    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-tls\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.335384 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.335363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xzg\" (UniqueName: \"kubernetes.io/projected/6aa3275d-c0c8-4ffb-9264-280b10d1672c-kube-api-access-t9xzg\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.339623 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.339547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddv27\" (UniqueName: \"kubernetes.io/projected/ac2fcf63-43d0-4759-9e77-edd3500ff388-kube-api-access-ddv27\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.340449 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.340428    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqfgx\" (UniqueName: \"kubernetes.io/projected/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-api-access-nqfgx\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.379852 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.379823    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:05.390057 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.390031    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wtrwm" event={"ID":"d613c8d4-0ecb-4178-839a-e626bd2646e5","Type":"ContainerStarted","Data":"f03d2523fff92b2dc272ef48c88c1a615f4990cbdab28aa4a8f34e39833cc96e"}
Nov 03 10:29:05.391331 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.391309    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d9x57" event={"ID":"254ba5e2-5f70-4980-9f35-705260e6b4a8","Type":"ContainerStarted","Data":"3e47fab9093886b012e2a6436c400d52895195cf1906a2c544d8545c05cb5b91"}
Nov 03 10:29:05.393303 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.393282    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jq8jl" event={"ID":"487f4942-ea5d-44a2-92dc-9fed886c9fdf","Type":"ContainerStarted","Data":"0c34f5a52e19f010a73acb56b95dbcd5d9ee1a5cf3d776be02c8cdc69e93a289"}
Nov 03 10:29:05.393399 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.393323    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jq8jl" event={"ID":"487f4942-ea5d-44a2-92dc-9fed886c9fdf","Type":"ContainerStarted","Data":"0ae770beb9391614b75861f167fdc3af4f5ff19b83c2df689e519c8bc25902ed"}
Nov 03 10:29:05.393399 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.393341    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jq8jl" event={"ID":"487f4942-ea5d-44a2-92dc-9fed886c9fdf","Type":"ContainerStarted","Data":"d4fe728f134e0fdee62072b73b006a18cc2a5f5288b6f86df64dbd3dcdc765e7"}
Nov 03 10:29:05.499724 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.499526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:05.510665 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.510641    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:05.511658 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.511630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:05.649704 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.649670    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:05.784140 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.784108    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"]
Nov 03 10:29:05.787390 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:05.787363    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1c1f57_6b4b_4cd9_9245_0d71ef61b0f2.slice/crio-d22e902e725870996bbcbbfb764822114c7e343bda64364c9be8931c7ccf6ec0 WatchSource:0}: Error finding container d22e902e725870996bbcbbfb764822114c7e343bda64364c9be8931c7ccf6ec0: Status 404 returned error can't find the container with id d22e902e725870996bbcbbfb764822114c7e343bda64364c9be8931c7ccf6ec0
Nov 03 10:29:05.793851 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.793825    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:05.826888 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.826823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:05.829949 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:05.829923    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:06.083613 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.083486    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:06.253193 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.253155    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:06.384004 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.383704    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"]
Nov 03 10:29:06.387738 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.387524    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:06.391389 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:06.391350    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2fcf63_43d0_4759_9e77_edd3500ff388.slice/crio-9acd3a44bfeb6da7231f277739838911133859045c1f02051a648081ac6b747a WatchSource:0}: Error finding container 9acd3a44bfeb6da7231f277739838911133859045c1f02051a648081ac6b747a: Status 404 returned error can't find the container with id 9acd3a44bfeb6da7231f277739838911133859045c1f02051a648081ac6b747a
Nov 03 10:29:06.398861 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.398834    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68" event={"ID":"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2","Type":"ContainerStarted","Data":"d22e902e725870996bbcbbfb764822114c7e343bda64364c9be8931c7ccf6ec0"}
Nov 03 10:29:06.400275 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.400250    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz" event={"ID":"ac2fcf63-43d0-4759-9e77-edd3500ff388","Type":"ContainerStarted","Data":"9acd3a44bfeb6da7231f277739838911133859045c1f02051a648081ac6b747a"}
Nov 03 10:29:06.404039 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.404014    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpkfs" event={"ID":"6aa3275d-c0c8-4ffb-9264-280b10d1672c","Type":"ContainerStarted","Data":"5d9b14ab60db68f4ac3bb5c36601a22207999a6d40df73fdb63600f06b1476cb"}
Nov 03 10:29:06.551015 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.550989    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:06.641701 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.641621    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxvr7_3bf53482-bc2a-4f95-a0ec-d2064baeb4f6/node-ca/0.log"
Nov 03 10:29:06.906705 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:06.906352    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:07.034431 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.034395    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:07.413398 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.413350    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz" event={"ID":"ac2fcf63-43d0-4759-9e77-edd3500ff388","Type":"ContainerStarted","Data":"5d0308c3d4bf4c72a83398d21e0029006dbc0f7f25d8d5049649ecec0f557052"}
Nov 03 10:29:07.414081 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.413458    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz" event={"ID":"ac2fcf63-43d0-4759-9e77-edd3500ff388","Type":"ContainerStarted","Data":"d4ac9ac3b156aec3c6b5770817d57a2085f9f3cef59b977192bc4e31d3b194a5"}
Nov 03 10:29:07.416550 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.416478    2576 generic.go:356] "Generic (PLEG): container finished" podID="6aa3275d-c0c8-4ffb-9264-280b10d1672c" containerID="49036025d50bf7eb1f7300f2fc278bca315dccd3deaed28103e5debeda368e7c" exitCode=0
Nov 03 10:29:07.416670 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.416594    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpkfs" event={"ID":"6aa3275d-c0c8-4ffb-9264-280b10d1672c","Type":"ContainerDied","Data":"49036025d50bf7eb1f7300f2fc278bca315dccd3deaed28103e5debeda368e7c"}
Nov 03 10:29:07.421027 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.421002    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jq8jl" event={"ID":"487f4942-ea5d-44a2-92dc-9fed886c9fdf","Type":"ContainerStarted","Data":"42de1ac531ab7395fa8dcf266d35593f9e715395972f42a92fd98d3b0acf2a2f"}
Nov 03 10:29:07.424139 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.424113    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wtrwm" event={"ID":"d613c8d4-0ecb-4178-839a-e626bd2646e5","Type":"ContainerStarted","Data":"b0937752e9749d660047217f014ca64e30be088a9977397a6115ae7fc2eeff00"}
Nov 03 10:29:07.424232 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.424163    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wtrwm" event={"ID":"d613c8d4-0ecb-4178-839a-e626bd2646e5","Type":"ContainerStarted","Data":"9cf7dc192496421ebf213fa56ececd34562b39ed762c3a60082c939075ff0353"}
Nov 03 10:29:07.424673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.424653    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:07.449271 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.449249    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:07.454223 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.454128    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-wtrwm" podStartSLOduration=2.011137626 podStartE2EDuration="4.454091273s" totalImagesPullingTime="2.442953647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:07.452011559 +0000 UTC m=+49.849971191" watchObservedRunningTime="2026-11-03 10:29:07.454091273 +0000 UTC m=+49.852050940"
Nov 03 10:29:07.473325 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.472962    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jq8jl" podStartSLOduration=1.923783281 podStartE2EDuration="4.472945532s" totalImagesPullingTime="2.549162251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:29:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:07.471663899 +0000 UTC m=+49.869623531" watchObservedRunningTime="2026-11-03 10:29:07.472945532 +0000 UTC m=+49.870905163"
Nov 03 10:29:07.582521 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:07.582497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:08.138614 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.138513    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:08.270255 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.270225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:08.399948 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.399863    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:08.429228 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.429194    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68" event={"ID":"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2","Type":"ContainerStarted","Data":"311390b9f1a42a54cb2fa715e6340104115e96b397cdf4eb66f17337e1a6ad2c"}
Nov 03 10:29:08.429641 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.429253    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68" event={"ID":"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2","Type":"ContainerStarted","Data":"fb5e6c58a48e0de362aae71c53f498f1e5416c2ca6a8cdb55f607e87862d0b3f"}
Nov 03 10:29:08.429641 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.429283    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68" event={"ID":"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2","Type":"ContainerStarted","Data":"022f3204e449edccd64e27704e0b4ee7a7a2b9669dc7fe2773928fccd732b48d"}
Nov 03 10:29:08.431379 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.431349    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz" event={"ID":"ac2fcf63-43d0-4759-9e77-edd3500ff388","Type":"ContainerStarted","Data":"aa4fa10210483af22673d094fde60ee1351238e15c2b0b6678db164ec8169695"}
Nov 03 10:29:08.433975 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.433893    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpkfs" event={"ID":"6aa3275d-c0c8-4ffb-9264-280b10d1672c","Type":"ContainerStarted","Data":"1c6492ce13b6f3267c1cf691e8ae6e5a043e62692aa29fc8af639a00d926961f"}
Nov 03 10:29:08.433975 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.433943    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpkfs" event={"ID":"6aa3275d-c0c8-4ffb-9264-280b10d1672c","Type":"ContainerStarted","Data":"93c1a548c72d7e666d0be8fe2ce150eaa981115db4b277119b5c36cf22d04491"}
Nov 03 10:29:08.447772 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.447735    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68" podStartSLOduration=1.389560541 podStartE2EDuration="3.447724218s" totalImagesPullingTime="2.058163677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:08.447344284 +0000 UTC m=+50.845303918" watchObservedRunningTime="2026-11-03 10:29:08.447724218 +0000 UTC m=+50.845683849"
Nov 03 10:29:08.468107 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.468066    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz" podStartSLOduration=2.172718318 podStartE2EDuration="3.468052522s" totalImagesPullingTime="1.295334204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:08.467469921 +0000 UTC m=+50.865429554" watchObservedRunningTime="2026-11-03 10:29:08.468052522 +0000 UTC m=+50.866012153"
Nov 03 10:29:08.518950 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:08.518908    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mpkfs" podStartSLOduration=2.20536936 podStartE2EDuration="3.518896794s" totalImagesPullingTime="1.225608497s" totalInitContainerRuntime="87.918937ms" isStatefulPod=true podCreationTimestamp="2026-11-03 10:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:08.518477842 +0000 UTC m=+50.916437477" watchObservedRunningTime="2026-11-03 10:29:08.518896794 +0000 UTC m=+50.916856427"
Nov 03 10:29:10.148148 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:10.148114    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:10.277465 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:10.277424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:10.651177 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:10.651146    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:10.780059 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:10.779983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:11.388588 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.388552    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:11.392137 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.392121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.401589 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.401546    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 10:29:11.401935 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.401920    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 10:29:11.402174 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.402153    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 10:29:11.402268 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.402193    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 10:29:11.402268 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.402155    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d9v2v0thtuv8v\""
Nov 03 10:29:11.402268 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.402211    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 10:29:11.402268 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.402228    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 10:29:11.403876 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.403856    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 10:29:11.403876 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.403871    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 10:29:11.404416 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.404297    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 10:29:11.404416 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.403868    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 10:29:11.404416 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.404352    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 10:29:11.404718 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.404695    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 10:29:11.404776 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.404731    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 10:29:11.408850 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.405041    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 10:29:11.408850 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.405354    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cb568\""
Nov 03 10:29:11.408850 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.408810    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 10:29:11.410507 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.410342    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:11.445068 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.445039    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d9x57" event={"ID":"254ba5e2-5f70-4980-9f35-705260e6b4a8","Type":"ContainerStarted","Data":"a6f5667226a0f5ec3a95c54fe2743f0ea1e2429d1ae1b74cbe3e5c247006d1ec"}
Nov 03 10:29:11.455425 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.455402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:11.459765 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.459729    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d9x57" podStartSLOduration=2.529520093 podStartE2EDuration="8.459718147s" totalImagesPullingTime="5.930198054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:11.459627015 +0000 UTC m=+53.857586648" watchObservedRunningTime="2026-11-03 10:29:11.459718147 +0000 UTC m=+53.857677778"
Nov 03 10:29:11.473615 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.473736 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.473736 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.473736 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.473857 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.473857 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.473928 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.473928 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.473928 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474068 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.473937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474068 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474258 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474258 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474258 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474258 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474455 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474455 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474304    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474455 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474455 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9br\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.474455 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.474443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575253 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575409 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575409 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575409 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575409 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575627 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575627 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575627 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575627 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575627 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575627 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.575889 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.575761    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.576603 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.576375    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.576603 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.576455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.576603 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.576487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.576603 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.576523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.576603 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.576547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.577105 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.576609    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.577593 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.577180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.577593 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.577222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.577593 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.577251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.577593 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.577295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9br\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.577593 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.577352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.577593 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.577463    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.578261 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.578237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.580179 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.580156    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.580294 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.580192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.580455 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.580435    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.580698 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.580671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.580854 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.580677    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.581079 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.581059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.581504 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.581477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.581504 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.581496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.581673 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.581648    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.581996 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.581978    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.582448 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.582429    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.583310 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.583292    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.583483 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.583463    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.583602 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.583557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.587636 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.587566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9br\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.721020 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.720999    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:11.734766 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.734740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:11.859739 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:11.859711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:12.003495 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:12.003425    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:12.006969 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:12.006943    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d34fc88_a058_4770_b5e7_dea808286e2d.slice/crio-1eda2a1032d1a83c5918a766342e4640d6ec22344a790703393df474b308f048 WatchSource:0}: Error finding container 1eda2a1032d1a83c5918a766342e4640d6ec22344a790703393df474b308f048: Status 404 returned error can't find the container with id 1eda2a1032d1a83c5918a766342e4640d6ec22344a790703393df474b308f048
Nov 03 10:29:12.105370 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:12.105345    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:12.242934 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:12.242904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:12.352009 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:12.351960    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:12.449304 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:12.449282    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerStarted","Data":"1eda2a1032d1a83c5918a766342e4640d6ec22344a790703393df474b308f048"}
Nov 03 10:29:12.539287 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:12.539264    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:12.669393 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:12.669336    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:13.143163 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:13.143126    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:13.398261 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:13.398177    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:14.457442 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:14.457411    2576 generic.go:356] "Generic (PLEG): container finished" podID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerID="8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d" exitCode=0
Nov 03 10:29:14.457818 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:14.457480    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerDied","Data":"8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d"}
Nov 03 10:29:18.439670 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:18.439633    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:18.474222 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:18.474173    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerStarted","Data":"46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6"}
Nov 03 10:29:18.474379 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:18.474253    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerStarted","Data":"222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474"}
Nov 03 10:29:19.174546 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:19.174516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:19.328267 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:19.328149    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:20.483901 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:20.483868    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerStarted","Data":"e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae"}
Nov 03 10:29:20.484377 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:20.483936    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerStarted","Data":"d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8"}
Nov 03 10:29:20.484377 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:20.483960    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerStarted","Data":"f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6"}
Nov 03 10:29:20.484377 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:20.483985    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerStarted","Data":"009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d"}
Nov 03 10:29:20.513018 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:20.512939    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.469936594 podStartE2EDuration="9.512926485s" totalImagesPullingTime="7.025623425s" totalInitContainerRuntime="17.366466ms" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:20.511160504 +0000 UTC m=+62.909120136" watchObservedRunningTime="2026-11-03 10:29:20.512926485 +0000 UTC m=+62.910886115"
Nov 03 10:29:20.573083 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:20.573052    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:20.707466 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:20.707432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:21.721131 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:21.721100    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:23.195609 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.195558    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:23.315950 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.315917    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:23.792846 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.792797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:23.795532 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.795510    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 10:29:23.805897 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.805874    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 10:29:23.816496 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.816460    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:23.893851 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.893817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:23.896472 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.896454    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 10:29:23.906802 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.906780    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:23.994903 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.994855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:23.997441 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:23.997423    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 10:29:24.007722 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.007701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:24.031861 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.031837    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pthxs\""
Nov 03 10:29:24.038224 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.038200    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ssvsh\""
Nov 03 10:29:24.039238 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.039223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:24.043833 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.043819    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lq8vs\""
Nov 03 10:29:24.046892 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.046878    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:24.051808 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.051788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:24.324489 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.324465    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4qhrr"]
Nov 03 10:29:24.326098 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.326071    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-brkhs"]
Nov 03 10:29:24.326871 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:24.326848    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd31aaf2_4824_4357_818f_aaa8663c3917.slice/crio-476a01f5e0ff6e28b5400ad332b8a66d8de08001254210b49bf5673e72c7f7b0 WatchSource:0}: Error finding container 476a01f5e0ff6e28b5400ad332b8a66d8de08001254210b49bf5673e72c7f7b0: Status 404 returned error can't find the container with id 476a01f5e0ff6e28b5400ad332b8a66d8de08001254210b49bf5673e72c7f7b0
Nov 03 10:29:24.329136 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:24.329075    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7075e9e0_fca9_44ba_82bf_e93c702b576d.slice/crio-797ec6f1b718b17012edfeac42d352ade43b80fb1154187107c8d7eea975cfd2 WatchSource:0}: Error finding container 797ec6f1b718b17012edfeac42d352ade43b80fb1154187107c8d7eea975cfd2: Status 404 returned error can't find the container with id 797ec6f1b718b17012edfeac42d352ade43b80fb1154187107c8d7eea975cfd2
Nov 03 10:29:24.334272 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.334254    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zwj2x"]
Nov 03 10:29:24.337701 ip-10-0-138-233 kubenswrapper[2576]: W1103 10:29:24.337682    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1ffb3b_e837_4cb6_b2b2_b2c55771e14e.slice/crio-8efd9bc513a24549bd900282c046b98b288e016a1bcb15003ef484730cf3e589 WatchSource:0}: Error finding container 8efd9bc513a24549bd900282c046b98b288e016a1bcb15003ef484730cf3e589: Status 404 returned error can't find the container with id 8efd9bc513a24549bd900282c046b98b288e016a1bcb15003ef484730cf3e589
Nov 03 10:29:24.395710 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.395683    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:24.496597 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.496551    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-brkhs" event={"ID":"7075e9e0-fca9-44ba-82bf-e93c702b576d","Type":"ContainerStarted","Data":"797ec6f1b718b17012edfeac42d352ade43b80fb1154187107c8d7eea975cfd2"}
Nov 03 10:29:24.497632 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.497598    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4qhrr" event={"ID":"cd31aaf2-4824-4357-818f-aaa8663c3917","Type":"ContainerStarted","Data":"476a01f5e0ff6e28b5400ad332b8a66d8de08001254210b49bf5673e72c7f7b0"}
Nov 03 10:29:24.498597 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.498552    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zwj2x" event={"ID":"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e","Type":"ContainerStarted","Data":"8efd9bc513a24549bd900282c046b98b288e016a1bcb15003ef484730cf3e589"}
Nov 03 10:29:24.529132 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:24.529104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:26.509370 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:26.509332    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4qhrr" event={"ID":"cd31aaf2-4824-4357-818f-aaa8663c3917","Type":"ContainerStarted","Data":"a237a41dbb7e976459e13c263fb02ac0ee7c3f37d5e1348bac82c13bb3f48ae6"}
Nov 03 10:29:26.509760 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:26.509408    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4qhrr" event={"ID":"cd31aaf2-4824-4357-818f-aaa8663c3917","Type":"ContainerStarted","Data":"1add8b570df3852cf45e7651fdade7fcdac07c14e0121e0f55dc45f9980aece4"}
Nov 03 10:29:26.528541 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:26.528490    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4qhrr" podStartSLOduration=66.217000796 podStartE2EDuration="1m7.52847593s" totalImagesPullingTime="1.311475134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:26.526137136 +0000 UTC m=+68.924096769" watchObservedRunningTime="2026-11-03 10:29:26.52847593 +0000 UTC m=+68.926435561"
Nov 03 10:29:26.682779 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:26.682738    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:26.808897 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:26.808813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:27.003597 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:27.003549    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:27.137945 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:27.137872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:27.301712 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:27.301646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:27.431531 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:27.431502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:28.518532 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:28.518489    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-brkhs" event={"ID":"7075e9e0-fca9-44ba-82bf-e93c702b576d","Type":"ContainerStarted","Data":"17f8c952bf8b2831e8adabf3492ed54c0e37cbe84783a516cb8b087c35b76436"}
Nov 03 10:29:28.519021 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:28.518636    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:28.534503 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:28.534454    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-brkhs" podStartSLOduration=67.182074879 podStartE2EDuration="1m10.534439567s" totalImagesPullingTime="3.352364688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:28.533070497 +0000 UTC m=+70.931030132" watchObservedRunningTime="2026-11-03 10:29:28.534439567 +0000 UTC m=+70.932399200"
Nov 03 10:29:29.738516 ip-10-0-138-233 kubenswrapper[2576]: I1103 10:29:29.738348    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:29:29.738499 ip-10-0-138-233 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 10:29:29.747978 ip-10-0-138-233 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 10:29:29.748148 ip-10-0-138-233 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 10:29:29.748298 ip-10-0-138-233 systemd[1]: kubelet.service: Consumed 3.994s CPU time.
Nov 03 10:29:29.764163 ip-10-0-138-233 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 10:29:29.823992 ip-10-0-138-233 kubenswrapper[5817]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:29:29.823992 ip-10-0-138-233 kubenswrapper[5817]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 10:29:29.823992 ip-10-0-138-233 kubenswrapper[5817]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:29:29.823992 ip-10-0-138-233 kubenswrapper[5817]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:29:29.825986 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.825917    5817 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:29.825986 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.825986    5817 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.825990    5817 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.825994    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.825996    5817 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.825999    5817 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826002    5817 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826005    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826008    5817 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826010    5817 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826013    5817 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826015    5817 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826018    5817 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826020    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826023    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826025    5817 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826028    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826030    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826033    5817 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826035    5817 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826038    5817 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:29.826049 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826040    5817 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826043    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826045    5817 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826048    5817 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826051    5817 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826058    5817 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826060    5817 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826063    5817 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826066    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826068    5817 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826071    5817 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826073    5817 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826076    5817 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826078    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826080    5817 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826083    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826085    5817 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826088    5817 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826090    5817 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:29.826540 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826093    5817 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826095    5817 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826097    5817 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826100    5817 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826102    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826104    5817 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826107    5817 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826109    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826111    5817 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826114    5817 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826116    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826119    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826122    5817 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826124    5817 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826127    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826129    5817 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826132    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826135    5817 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826138    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826140    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:29.827008 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826143    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826145    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826148    5817 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826150    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826152    5817 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826155    5817 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826157    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826159    5817 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826166    5817 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826168    5817 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826171    5817 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826175    5817 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826178    5817 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826182    5817 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826185    5817 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826188    5817 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826191    5817 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826193    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826197    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826199    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:29.827542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826202    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826205    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826207    5817 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826209    5817 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826212    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826214    5817 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826217    5817 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826219    5817 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826222    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826224    5817 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826233    5817 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826237    5817 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826241    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826244    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826247    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826249    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826253    5817 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826255    5817 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826258    5817 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:29.828054 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826261    5817 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826263    5817 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826266    5817 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826268    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826271    5817 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826274    5817 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.826276    5817 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827049    5817 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827055    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827059    5817 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827063    5817 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827068    5817 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827071    5817 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827074    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827077    5817 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827079    5817 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827082    5817 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827084    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827087    5817 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:29.828542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827089    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827092    5817 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827095    5817 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827098    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827100    5817 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827111    5817 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827114    5817 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827116    5817 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827119    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827122    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827124    5817 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827127    5817 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827129    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827132    5817 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827134    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827137    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827139    5817 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827142    5817 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827144    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827147    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:29.829034 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827149    5817 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827152    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827155    5817 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827158    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827160    5817 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827163    5817 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827165    5817 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827168    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827170    5817 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827173    5817 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827175    5817 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827178    5817 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827180    5817 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827182    5817 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827186    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827188    5817 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827191    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827193    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827201    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:29.829624 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827204    5817 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827207    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827209    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827211    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827214    5817 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827216    5817 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827219    5817 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827221    5817 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827224    5817 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827226    5817 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827229    5817 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827231    5817 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827234    5817 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827237    5817 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827239    5817 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827242    5817 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827244    5817 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827247    5817 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827249    5817 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827252    5817 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:29.830077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827255    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827257    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827260    5817 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827263    5817 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827267    5817 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827270    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827273    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827276    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827279    5817 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827282    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827284    5817 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827287    5817 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827295    5817 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827298    5817 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827300    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827303    5817 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827305    5817 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827307    5817 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827310    5817 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:29.830560 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827312    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827315    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827317    5817 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827320    5817 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827322    5817 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827325    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827328    5817 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827330    5817 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827333    5817 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827335    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827338    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827340    5817 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827342    5817 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827345    5817 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827351    5817 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827353    5817 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827892    5817 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827898    5817 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827901    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827904    5817 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:29.831035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827906    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827917    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827921    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827924    5817 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827927    5817 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827936    5817 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827939    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827942    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827945    5817 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827947    5817 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827950    5817 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827952    5817 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827955    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827957    5817 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827960    5817 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827963    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827966    5817 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827968    5817 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827970    5817 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827973    5817 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:29.831538 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827975    5817 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827978    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827980    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827983    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827985    5817 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827988    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827990    5817 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827993    5817 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827995    5817 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.827998    5817 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828002    5817 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828006    5817 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828009    5817 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828012    5817 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828014    5817 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828017    5817 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828019    5817 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828022    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828030    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:29.832080 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828032    5817 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828035    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828037    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828045    5817 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828048    5817 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828050    5817 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828053    5817 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828056    5817 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828059    5817 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828062    5817 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828065    5817 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828068    5817 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828071    5817 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828073    5817 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828076    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828078    5817 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828081    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828083    5817 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828086    5817 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:29.832539 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828088    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828091    5817 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828093    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828095    5817 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828098    5817 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828102    5817 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828106    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828109    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828112    5817 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828115    5817 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828117    5817 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828119    5817 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828122    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828130    5817 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828133    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828136    5817 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828138    5817 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828141    5817 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828143    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:29.833035 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828146    5817 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828148    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828151    5817 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828153    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828156    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828158    5817 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828161    5817 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828163    5817 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828166    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828169    5817 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828171    5817 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828173    5817 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828176    5817 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828178    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828180    5817 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828183    5817 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828185    5817 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828190    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828193    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828196    5817 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:29.833481 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828198    5817 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828201    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828203    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828205    5817 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.828208    5817 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828313    5817 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828319    5817 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828346    5817 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828355    5817 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828359    5817 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828362    5817 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828369    5817 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828373    5817 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828376    5817 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828379    5817 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828382    5817 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828385    5817 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828388    5817 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828391    5817 flags.go:64] FLAG: --cgroup-root=""
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828394    5817 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828397    5817 flags.go:64] FLAG: --client-ca-file=""
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828400    5817 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828403    5817 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828409    5817 flags.go:64] FLAG: --cluster-domain=""
Nov 03 10:29:29.833976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828412    5817 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828415    5817 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828418    5817 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828421    5817 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828425    5817 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828429    5817 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828432    5817 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828435    5817 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828438    5817 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828441    5817 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828444    5817 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828447    5817 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828450    5817 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828454    5817 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828457    5817 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828460    5817 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828470    5817 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828474    5817 flags.go:64] FLAG: --enable-server="true"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828477    5817 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828484    5817 flags.go:64] FLAG: --event-burst="100"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828487    5817 flags.go:64] FLAG: --event-qps="50"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828489    5817 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828493    5817 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828495    5817 flags.go:64] FLAG: --eviction-hard=""
Nov 03 10:29:29.834631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828499    5817 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828502    5817 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828505    5817 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828508    5817 flags.go:64] FLAG: --eviction-soft=""
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828511    5817 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828514    5817 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828517    5817 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828520    5817 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828523    5817 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828526    5817 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828528    5817 flags.go:64] FLAG: --feature-gates=""
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828532    5817 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828535    5817 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828540    5817 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828543    5817 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828546    5817 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828552    5817 flags.go:64] FLAG: --help="false"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828555    5817 flags.go:64] FLAG: --hostname-override="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828558    5817 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828561    5817 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828564    5817 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828581    5817 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828585    5817 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 10:29:29.835614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828587    5817 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828590    5817 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828606    5817 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828609    5817 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828612    5817 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828615    5817 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828618    5817 flags.go:64] FLAG: --kube-reserved=""
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828621    5817 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828624    5817 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828627    5817 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828630    5817 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828632    5817 flags.go:64] FLAG: --lock-file=""
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828635    5817 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828638    5817 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828641    5817 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828647    5817 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828650    5817 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828653    5817 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828655    5817 flags.go:64] FLAG: --logging-format="text"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828658    5817 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828661    5817 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828664    5817 flags.go:64] FLAG: --manifest-url=""
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828668    5817 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828672    5817 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828675    5817 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 10:29:29.836244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828681    5817 flags.go:64] FLAG: --max-pods="110"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828684    5817 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828686    5817 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828689    5817 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828692    5817 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828695    5817 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828698    5817 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828700    5817 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828706    5817 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828709    5817 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828718    5817 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828722    5817 flags.go:64] FLAG: --pod-cidr=""
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828725    5817 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828728    5817 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828731    5817 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828734    5817 flags.go:64] FLAG: --port="10250"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828737    5817 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828739    5817 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02ea222e3a0053047"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828742    5817 flags.go:64] FLAG: --qos-reserved=""
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828745    5817 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828748    5817 flags.go:64] FLAG: --register-node="true"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828751    5817 flags.go:64] FLAG: --register-with-taints=""
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828755    5817 flags.go:64] FLAG: --registry-burst="10"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828758    5817 flags.go:64] FLAG: --registry-qps="5"
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828760    5817 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828763    5817 flags.go:64] FLAG: --reserved-memory=""
Nov 03 10:29:29.836923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828767    5817 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828770    5817 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828773    5817 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828777    5817 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828780    5817 flags.go:64] FLAG: --runonce="false"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828783    5817 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828787    5817 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828790    5817 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828792    5817 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828795    5817 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828798    5817 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828801    5817 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828804    5817 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828807    5817 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828810    5817 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828812    5817 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828815    5817 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828823    5817 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828826    5817 flags.go:64] FLAG: --system-cgroups=""
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828829    5817 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828834    5817 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828837    5817 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828840    5817 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828846    5817 flags.go:64] FLAG: --tls-min-version=""
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828849    5817 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 10:29:29.837730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828851    5817 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828854    5817 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828857    5817 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828860    5817 flags.go:64] FLAG: --v="2"
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828864    5817 flags.go:64] FLAG: --version="false"
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828868    5817 flags.go:64] FLAG: --vmodule=""
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828872    5817 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.828875    5817 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829030    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829033    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829038    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829041    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829044    5817 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829048    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829050    5817 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829053    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829055    5817 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829058    5817 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829060    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829063    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829066    5817 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:29.838319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829068    5817 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829070    5817 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829073    5817 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829075    5817 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829078    5817 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829081    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829083    5817 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829086    5817 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829088    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829090    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829093    5817 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829095    5817 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829097    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829100    5817 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829102    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829104    5817 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829107    5817 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829109    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829112    5817 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829115    5817 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:29.838898 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829117    5817 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829120    5817 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829123    5817 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829125    5817 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829128    5817 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829130    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829133    5817 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829135    5817 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829138    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829140    5817 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829143    5817 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829145    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829148    5817 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829150    5817 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829154    5817 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829158    5817 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829161    5817 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829164    5817 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829167    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829169    5817 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:29.839509 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829172    5817 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829174    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829176    5817 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829179    5817 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829181    5817 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829184    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829186    5817 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829188    5817 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829191    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829193    5817 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829196    5817 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829198    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829201    5817 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829204    5817 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829206    5817 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829209    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829212    5817 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829214    5817 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829217    5817 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:29.840245 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829219    5817 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829222    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829225    5817 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829229    5817 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829232    5817 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829235    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829237    5817 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829240    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829242    5817 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829245    5817 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829247    5817 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829250    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829252    5817 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829255    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829257    5817 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829260    5817 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829263    5817 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829265    5817 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829268    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:29.840879 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829270    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829272    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829275    5817 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829277    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829280    5817 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829283    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829285    5817 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829287    5817 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829290    5817 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829292    5817 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829294    5817 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829297    5817 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829299    5817 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829302    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829304    5817 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.829331    5817 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:29:29.841333 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829396    5817 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829400    5817 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829403    5817 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829406    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829408    5817 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829411    5817 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829413    5817 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829416    5817 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829418    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829423    5817 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829426    5817 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829430    5817 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829434    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829437    5817 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829441    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829444    5817 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829447    5817 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829449    5817 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829452    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:29.841729 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829454    5817 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829457    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829459    5817 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829462    5817 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829464    5817 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829467    5817 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829469    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829472    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829474    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829477    5817 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829479    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829482    5817 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829484    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829487    5817 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829489    5817 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829491    5817 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829494    5817 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829496    5817 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829498    5817 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829501    5817 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:29.842332 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829503    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829506    5817 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829508    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829511    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829514    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829518    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829520    5817 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829524    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829526    5817 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829529    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829531    5817 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829534    5817 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829536    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829540    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829543    5817 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829545    5817 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829548    5817 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829550    5817 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829552    5817 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:29.843069 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829555    5817 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829557    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829560    5817 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829562    5817 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829565    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829581    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829584    5817 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829586    5817 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829588    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829591    5817 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829594    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829596    5817 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829599    5817 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829601    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829604    5817 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829606    5817 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829609    5817 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829612    5817 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829615    5817 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829618    5817 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:29.843534 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829622    5817 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829624    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829627    5817 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829630    5817 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829632    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829635    5817 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829637    5817 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829640    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829642    5817 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829645    5817 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829648    5817 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829651    5817 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829655    5817 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829657    5817 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829660    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829663    5817 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829665    5817 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829668    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829670    5817 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829673    5817 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:29.844031 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829675    5817 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829678    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829680    5817 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829683    5817 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829685    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829688    5817 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829690    5817 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.829693    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.833286    5817 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.833301    5817 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.833314    5817 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.833319    5817 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833372    5817 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833377    5817 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833380    5817 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833383    5817 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833386    5817 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833388    5817 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833392    5817 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:29.844507 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833394    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833397    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833400    5817 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833402    5817 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833406    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833409    5817 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833411    5817 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833414    5817 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833416    5817 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833419    5817 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833421    5817 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833424    5817 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833426    5817 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833429    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833432    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833434    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833437    5817 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833440    5817 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833443    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833445    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:29.844978 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833448    5817 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833450    5817 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833453    5817 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833456    5817 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833458    5817 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833461    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833464    5817 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833466    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833469    5817 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833471    5817 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833474    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833476    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833479    5817 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833481    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833484    5817 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833486    5817 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833489    5817 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833492    5817 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833495    5817 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833497    5817 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:29.845450 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833500    5817 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833502    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833505    5817 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833507    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833510    5817 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833512    5817 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833515    5817 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833518    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833520    5817 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833523    5817 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833526    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833529    5817 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833531    5817 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833533    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833537    5817 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833541    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833544    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833546    5817 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833549    5817 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:29.845937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833551    5817 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833554    5817 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833556    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833559    5817 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833561    5817 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833564    5817 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833566    5817 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833587    5817 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833592    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833596    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833599    5817 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833602    5817 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833605    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833608    5817 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833610    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833613    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833615    5817 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833618    5817 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833621    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:29.846424 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833623    5817 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833625    5817 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833628    5817 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833631    5817 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833633    5817 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833636    5817 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833639    5817 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833641    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833643    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833646    5817 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833648    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833651    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833653    5817 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833656    5817 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833658    5817 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833660    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833663    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833665    5817 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833668    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:29.846897 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833670    5817 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833672    5817 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.833699    5817 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833832    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833837    5817 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833840    5817 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833843    5817 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833846    5817 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833848    5817 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833851    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833853    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833856    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833858    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833861    5817 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833863    5817 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833865    5817 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:29.847338 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833868    5817 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833871    5817 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833873    5817 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833876    5817 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833878    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833880    5817 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833883    5817 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833886    5817 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833888    5817 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833890    5817 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833893    5817 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833895    5817 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833898    5817 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833900    5817 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833903    5817 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833905    5817 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833908    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833910    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833912    5817 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833915    5817 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:29.847736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833918    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833920    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833922    5817 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833925    5817 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833928    5817 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833930    5817 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833933    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833935    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833938    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833940    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833942    5817 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833945    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833948    5817 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833950    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833953    5817 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833955    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833957    5817 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833960    5817 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833962    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:29.848224 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833965    5817 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833967    5817 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833970    5817 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833972    5817 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833975    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833977    5817 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833980    5817 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833982    5817 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833986    5817 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833990    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833993    5817 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833995    5817 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.833998    5817 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834000    5817 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834003    5817 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834005    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834008    5817 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834010    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834013    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834015    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:29.848725 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834018    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834020    5817 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834022    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834025    5817 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834028    5817 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834030    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834033    5817 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834035    5817 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834037    5817 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834040    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834043    5817 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834047    5817 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834049    5817 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834052    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834054    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834057    5817 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834059    5817 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834061    5817 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834064    5817 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:29.849202 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834066    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834072    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834074    5817 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834077    5817 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834079    5817 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834082    5817 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834084    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834087    5817 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834089    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834092    5817 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834094    5817 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834097    5817 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834100    5817 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834102    5817 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834105    5817 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.834127    5817 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834158    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:29.849671 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834162    5817 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834165    5817 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834168    5817 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834172    5817 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834176    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834179    5817 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834182    5817 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834185    5817 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834187    5817 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834190    5817 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834192    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834196    5817 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834198    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834201    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834203    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834206    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834208    5817 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834212    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834214    5817 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:29.850083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834217    5817 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834220    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834222    5817 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834224    5817 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834227    5817 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834229    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834233    5817 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834237    5817 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834240    5817 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834243    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834245    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834248    5817 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834250    5817 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834253    5817 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834255    5817 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834258    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834260    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834263    5817 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834265    5817 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:29.850533 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834268    5817 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834270    5817 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834273    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834276    5817 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834278    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834281    5817 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834283    5817 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834286    5817 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834288    5817 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834290    5817 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834293    5817 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834296    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834299    5817 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834302    5817 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834304    5817 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834307    5817 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834309    5817 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834312    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834315    5817 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834317    5817 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:29.851083 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834320    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834322    5817 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834325    5817 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834328    5817 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834331    5817 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834334    5817 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834337    5817 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834340    5817 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834342    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834345    5817 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834347    5817 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834354    5817 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834357    5817 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834359    5817 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834362    5817 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834364    5817 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834367    5817 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834369    5817 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834371    5817 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834374    5817 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:29.851553 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834376    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834379    5817 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834381    5817 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834385    5817 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834387    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834390    5817 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834392    5817 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834395    5817 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834397    5817 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834401    5817 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834403    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834406    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834408    5817 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834410    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834413    5817 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834416    5817 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834418    5817 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834422    5817 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834424    5817 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:29.852459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834427    5817 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834429    5817 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834432    5817 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834434    5817 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834437    5817 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834439    5817 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834442    5817 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:29.834444    5817 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.834492    5817 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.836703    5817 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.836755    5817 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.836949    5817 server.go:1014] "Starting client certificate rotation"
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.837107    5817 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.837178    5817 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 10:23:18 +0000 UTC" deadline="2028-06-02 19:07:19.688444825 +0000 UTC"
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.837219    5817 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13856h37m49.851229768s"
Nov 03 10:29:29.853065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.837435    5817 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:29:29.853452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.838498    5817 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:29:29.853452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.839954    5817 log.go:25] "Validated CRI v1 runtime API"
Nov 03 10:29:29.853452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.841788    5817 log.go:25] "Validated CRI v1 image API"
Nov 03 10:29:29.853452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.842488    5817 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 10:29:29.853452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.844310    5817 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a5f97d21-a382-4edb-9d93-858f82c22150:/dev/nvme0n1p3 c3583c8b-86ad-4901-bdaa-949b653389d0:/dev/nvme0n1p4]
Nov 03 10:29:29.853626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.844331    5817 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c101fb3f12fd969652b2400b34f924547f88df44aee188c0a5dd32ba347f492/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c101fb3f12fd969652b2400b34f924547f88df44aee188c0a5dd32ba347f492/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1eda2a1032d1a83c5918a766342e4640d6ec22344a790703393df474b308f048/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1eda2a1032d1a83c5918a766342e4640d6ec22344a790703393df474b308f048/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/299b108faa17dcf3a53dedb8a4ce351eb7ba8cbad39ad2c039d631c363be1e0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/299b108faa17dcf3a53dedb8a4ce351eb7ba8cbad39ad2c039d631c363be1e0a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b9127068ef1d8b3ea9a2d4c74100d8b6ef16b6b1759a53bcab04ebb20a97bde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b9127068ef1d8b3ea9a2d4c74100d8b6ef16b6b1759a53bcab04ebb20a97bde/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e47fab9093886b012e2a6436c400d52895195cf1906a2c544d8545c05cb5b91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e47fab9093886b012e2a6436c400d52895195cf1906a2c544d8545c05cb5b91/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46e383e26addea4fc9566f042bf9f8e95e05d495fa7408e93df367a4ac593630/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46e383e26addea4fc9566f042bf9f8e95e05d495fa7408e93df367a4ac593630/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/476a01f5e0ff6e28b5400ad332b8a66d8de08001254210b49bf5673e72c7f7b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/476a01f5e0ff6e28b5400ad332b8a66d8de08001254210b49bf5673e72c7f7b0/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d9b14ab60db68f4ac3bb5c36601a22207999a6d40df73fdb63600f06b1476cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d9b14ab60db68f4ac3bb5c36601a22207999a6d40df73fdb63600f06b1476cb/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/640c3fc7954e29e1655ba46d9513b818a199820e185329f86fc11fadc3a5ebfb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/640c3fc7954e29e1655ba46d9513b818a199820e185329f86fc11fadc3a5ebfb/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/797ec6f1b718b17012edfeac42d352ade43b80fb1154187107c8d7eea975cfd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/797ec6f1b718b17012edfeac42d352ade43b80fb1154187107c8d7eea975cfd2/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/798e1f748fdb354fe892013fe551571574303c88599c06b8b117a3cbb6d91c0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/798e1f748fdb354fe892013fe551571574303c88599c06b8b117a3cbb6d91c0b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79e5efb821ab03a9dd347a84a09cf6686dadd40e44c445133bcc9d0217792f43/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79e5efb821ab03a9dd347a84a09cf6686dadd40e44c445133bcc9d0217792f43/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b48d395a1c66af09afa630d1d4c3b1173086b291478d651a40d154bc762f34c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b48d395a1c66af09afa630d1d4c3b1173086b291478d651a40d154bc762f34c/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8efd9bc513a24549bd900282c046b98b288e016a1bcb15003ef484730cf3e589/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8efd9bc513a24549bd900282c046b98b288e016a1bcb15003ef484730cf3e589/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97e54fd23d3668217337c906798e1e4c6183940adf814521684ba405c97b189f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97e54fd23d3668217337c906798e1e4c6183940adf814521684ba405c97b189f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9acd3a44bfeb6da7231f277739838911133859045c1f02051a648081ac6b747a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9acd3a44bfeb6da7231f277739838911133859045c1f02051a648081ac6b747a/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be6522254cd3b3401be94428e8dad0893e20b4e599345b87ccef0f1c48e337fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be6522254cd3b3401be94428e8dad0893e20b4e599345b87ccef0f1c48e337fd/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d22e902e725870996bbcbbfb764822114c7e343bda64364c9be8931c7ccf6ec0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d22e902e725870996bbcbbfb764822114c7e343bda64364c9be8931c7ccf6ec0/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4fe728f134e0fdee62072b73b006a18cc2a5f5288b6f86df64dbd3dcdc765e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4fe728f134e0fdee62072b73b006a18cc2a5f5288b6f86df64dbd3dcdc765e7/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f03d2523fff92b2dc272ef48c88c1a615f4990cbdab28aa4a8f34e39833cc96e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f03d2523fff92b2dc272ef48c88c1a615f4990cbdab28aa4a8f34e39833cc96e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0174417f-2cb0-4933-b9cc-7b044e2e8254/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0174417f-2cb0-4933-b9cc-7b044e2e8254/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/254ba5e2-5f70-4980-9f35-705260e6b4a8/volumes/kubernetes.io~projected/kube-api-access-fw56c:{mountpoint:/var/lib/kubelet/pods/254ba5e2-5f70-4980-9f35-705260e6b4a8/volumes/kubernetes.io~projected/kube-api-access-fw56c major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/254ba5e2-5f70-4980-9f35-705260e6b4a8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/254ba5e2-5f70-4980-9f35-705260e6b4a8/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/316f1ae7-9bef-4691-a73d-e79f01fdea14/volumes/kubernetes.io~projected/kube-api-access-rwp9g:{mountpoint:/var/lib/kubelet/pods/316f1ae7-9bef-4691-a73d-e79f01fdea14/volumes/kubernetes.io~projected/kube-api-access-rwp9g major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6/volumes/kubernetes.io~projected/kube-api-access-n9jxn:{mountpoint:/var/lib/kubelet/pods/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6/volumes/kubernetes.io~projected/kube-api-access-n9jxn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40cc057e-3d16-464d-8b2c-7a15d01238c1/volumes/kubernetes.io~projected/kube-api-access-jstbl:{mountpoint:/var/lib/kubelet/pods/40cc057e-3d16-464d-8b2c-7a15d01238c1/volumes/kubernetes.io~projected/kube-api-access-jstbl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/487f4942-ea5d-44a2-92dc-9fed886c9fdf/volumes/kubernetes.io~projected/kube-api-access-g2cbk:{mountpoint:/var/lib/kubelet/pods/487f4942-ea5d-44a2-92dc-9fed886c9fdf/volumes/kubernetes.io~projected/kube-api-access-g2cbk major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/487f4942-ea5d-44a2-92dc-9fed886c9fdf/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/487f4942-ea5d-44a2-92dc-9fed886c9fdf/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/565f923e-a384-496e-be31-f99ddaec54ce/volumes/kubernetes.io~projected/kube-api-access-h4c6f:{mountpoint:/var/lib/kubelet/pods/565f923e-a384-496e-be31-f99ddaec54ce/volumes/kubernetes.io~projected/kube-api-access-h4c6f major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e735188-d000-4d3e-9e65-f3320352f3b5/volumes/kubernetes.io~projected/kube-api-access-jrz7t:{mountpoint:/var/lib/kubelet/pods/5e735188-d000-4d3e-9e65-f3320352f3b5/volumes/kubernetes.io~projected/kube-api-access-jrz7t major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~projected/kube-api-access-t9xzg:{mountpoint:/var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~projected/kube-api-access-t9xzg major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volumes/kubernetes.io~projected/kube-api-access-d6v8d:{mountpoint:/var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volumes/kubernetes.io~projected/kube-api-access-d6v8d major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7075e9e0-fca9-44ba-82bf-e93c702b576d/volumes/kubernetes.io~projected/kube-api-access-mlg5g:{mountpoint:/var/lib/kubelet/pods/7075e9e0-fca9-44ba-82bf-e93c702b576d/volumes/kubernetes.io~projected/kube-api-access-mlg5g major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/config-reloader/2 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/prometheus/15 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~projected/kube-api-access-pl9br:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~projected/kube-api-access-pl9br major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~projected/tls-assets major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/config major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/web-config major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~projected/kube-api-access-cbmv4:{mountpoint:/var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~projected/kube-api-access-cbmv4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~projected/kube-api-access-ddv27:{mountpoint:/var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~projected/kube-api-access-ddv27 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96ecc89-8209-4f18-bc24-e96628147236/volumes/kubernetes.io~projected/kube-api-access-9ctqm:{mountpoint:/var/lib/kubelet/pods/b96ecc89-8209-4f18-bc24-e96628147236/volumes/kubernetes.io~projected/kube-api-access-9ctqm major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~projected/kube-api-access-nqfgx:{mountpoint:/var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~projected/kube-api-access-nqfgx major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd31aaf2-4824-4357-818f-aaa8663c3917/volumes/kubernetes.io~projected/kube-api-access-nbfqf:{mountpoint:/var/lib/kubelet/pods/cd31aaf2-4824-4357-818f-aaa8663c3917/volumes/kubernetes.io~projected/kube-api-access-nbfqf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd31aaf2-4824-4357-818f-aaa8663c3917/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/cd31aaf2-4824-4357-818f-aaa8663c3917/volumes/kubernetes.io~secret/metrics-certs major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d613c8d4-0ecb-4178-839a-e626bd2646e5/volumes/kubernetes.io~projected/kube-api-access-6vcnv:{mountpoint:/var/lib/kubelet/pods/d613c8d4-0ecb-4178-839a-e626bd2646e5/volumes/kubernetes.io~projected/kube-api-access-6vcnv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d613c8d4-0ecb-4178-839a-e626bd2646e5/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d613c8d4-0ecb-4178-839a-e626bd2646e5/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/84bb95b53179fa54e382d384e05cfcd09357cb3bf3a4e0702c7f39b56e9150f0/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b98f961f2817b11d08fb73a1e50ad7667c9495138ff8d76c914666b57f2fbf68/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2e17d41e990634fa97ebab3c437ec5c07e256dd405fd4331923c32d512cb2b85/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a1e71a0b3504be448a1e6ceb18f8846d913f9cf18ec80d069442b8741e4d2538/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e9d24cbbcacac5085df5c4c6e1c474e36c3091e6c851e422e42c0c86fae35f42/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8ce511e02a2446256fdef889bd1b0235ee4e4de4c664695b55102322b5023772/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4033d66e2da6a95d6b4b376b2d4dc41a568a7ca53dc643387bca5219da213786/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/88b6b60f78450f9bcdb50ad6984c8ddfa8b1dbf4d512abf9aef48c36f6618e49/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/eca50625b21b12f673676c12e31d159e7c5512f3950d8501d6d8f37f645c81a7/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9c3db0efeb09cd03f04946953c2903a68e2544809ca9ad0116657b15c93dbeb0/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/c27198fae7d18761ae2c440996d997766531c93a5a5d1f14f9f1c9c52d5db5ca/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/268fb37dd3c7da4ca01c8a82cd5932a2ad1fa47f7753300ee5d1a957d9ca7107/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/108869fab899ce41437b2cdb5dc1f7203ba0dbd34e4431a95e6b266136f43f83/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1113e0d6bc7c89f412614db8192a0ead204e5a3f017c6bd3ad625ea729504942/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b8308749e5b8224b8bd8a4fe473c6ba539ecfa046fcb36e92b3fb19928ef6fe9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7c541e4a8458b2b45914da179c61b691b17c21370c37659352cc06b836d3d1f0/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1556dac9c067848da758fa5203571a8e5e84bd5887803ddd13c482fce7bb587d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1f9bbe92c67e10921aa5fc6c7785224ff56db8198e77223621b3b5224da759a7/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/99aa465c45c5b85817e8b4d59af1bd9905ef28817f10f12c92ee333bae79e3db/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ece380b724c052a9c5ba43ee8b10ce1f6cf34c3accffde1ec197df227a9c3620/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6c89e66de89ee2d4bc74b7deb06663ec24103345199df3627453dd6032b00e1c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5543bbb4ccb56525548ab0f7ac57f248e77687f86ef205a275ba5437f25a094d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/da2a61b5bdd8e521c4ea4ffd66dbe6627cd4cd28d129b3d828051db02194ef2c/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/bc6db30ae3817cd807b83b4c7dd925ce7c11bd50102c7eb18bc8f1f372a190a6/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/29e80cb775082732c1e6f3fb32d9663be881d316f92fb4a4b2cc906c77f223f6/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/dd41e4380b1b6c731defcee2430da93e2b1e4ee720c558d20104f940c7f4cbb3/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/4230f1cd9e99276eaa1a728e2d0cd03cb37291e17e6f7563ef903bb3a80cd4b8/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/c5c474c7e23a964f4c3de83304e16b7b0e9b3a453872ad6652e9ef20789ec668/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/7b19dca4da6b3f51def7d76e0d49ba50c921aef328cd24af3daaee95cc143dd5/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/803888aca67f72fc90b320fdc9705439d5c7c60e64199fcd41d46da7e8b7acca/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/e44429f1024b7d0462a69caa24845104a3efabb04219c6f40ce3d827f1f014ea/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/290bdcb7d0e1f3f05b551ecd82adb82720374e51edd39aafd1013704221099e7/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/57bd19c38189f7e1955d95d284fcb94f107ccee08ed63beda166ed21365cad27/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/affe05c13647145c596c9f2e9c0c0030dbf2a423d081e17fed2f34b2fcfb29a6/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/cc1d3720531aa5e5b6fbdc92cfe325054e84a2d8e9a8b0ce0b7321853e8690ab/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/78515f2b1283b71bcaa270bd29a76f4d5dc5a69061d5102ac372081aea47e9f6/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/bdc816b3a6463249028f3ca55f40bcd5e4ab919f874928cceadf8c704fc31e12/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/0b43a9513edc5ed2ca0f39add258dbcc9b5c1962c2841ea5ffb0aea0ed87558b/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/c0d39c1b01cd5dce633e97f427ddb5ef0084de49d76e52af0c0cc84fa26fcd6c/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/2d73b0ef6ee172790da5261e298f706eb8b62617c5f167e4ba0de2fb1aa1d92c/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/67843bddd5aa49ec1ca9a5d2fdc9ebb212cfb80fb18a9711b6f558aebd9945c3/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/68e876f7e10b9bc0407ef3e061f2cdf625054499579ed520e80bcd91498d329a/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/165bf56e34c8ddeac39d99cf413cfab57c63808502c420c5477db2be7b4a1675/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/84fd61ef313f73bdca1f02e76dd82a84ad79e5eeea367e1376e2cab7d2eb8cee/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/673927a71446e2bd047802e541eb8b2ed4e97ba3477c30f7c7fdd674a5ef55d9/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/c88ea1154134da8e36b12fd060b52dc0a5bdf2fe4d6876788af76104be27fdac/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/9d7fb6da76f70087087b0a567d5a197a0e574bfde0f993902ea31c98bc484034/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/4e6d37cd1b189e38f5b150872cced84656f0177173d43fe179b3b3de50aadd63/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/8eb1cabd8c76185d3145fa9096ac59aac8e5c7c1a8ed4145c041b2835a396a85/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/421e6f5d4ca04d10c80907f37aeb01449f772368e4231b0f4748f7a9364ca378/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/9c049042ca8c5d88849503fd1e8b9cc6a7e01393c50443fe9314dda999d07633/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/06ba493cfc524c799de6086a6f3b6cde500cf5bfbe0c3a2b42013549e7fcfedd/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/c293724e9af1e6155cc9786e6a8faa695d93ee783ab62e071a98c6e86316ee1b/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/3cc727313ce32edd70e57923ec3e15be56299850d031a05ce7af3f8173e390f8/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/a016640b55d4b79af8232da3239ca0cf07915815bcfd43a99549ebfdfc6384a0/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/94849bac82d75949a2f5e161ea2120f5bfce9a1b100eabd164336382da0a4e8e/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/642a04faf83e471910519d2fcbe6dcd2330ba27354344936382bb8b5bfc831d7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/76fec2e552f7509f56adc78f5cbf22bc81b269584d582a001f79b9023981070e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bbd26492ad62159bd05fe3dab83d6f43af78d8d86d2532d4a06ef5ad9f5d54eb/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/7cb73998a07f5ecaaec43d8dc03ec0de39cdbc9bfcda511461514d07e032eac4/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/96afd138b1760e4245189c56d835c800ba7f93cc5d722a52ba039feb33659553/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/f1d92a3a9475d9657732d9b12dc86e6c7b26b7b09b851e7ce117d2b8006d39f4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/bfd038da7049da0dead8f618713e14099ad2df43c0135480890fce8aa968f2ab/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/28f04a04421e3fccfb3b7e576e817fcfda1f1fbee19b63e9b644dd4647a63735/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/94871953eded309318cf068483a344119fe8bb3989e6fdaf05ca6340d8b55ac0/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.851090    5817 manager.go:217] Machine: {Timestamp:2026-11-03 10:29:29.850638031 +0000 UTC m=+0.056220074 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094046 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27202688a9a1e27e73ef4d0b7cddf5 SystemUUID:ec272026-88a9-a1e2-7e73-ef4d0b7cddf5 BootID:f9dbc59b-a26e-44fb-9a86-88fc687e9664 Filesystems:[{Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c101fb3f12fd969652b2400b34f924547f88df44aee188c0a5dd32ba347f492/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b9127068ef1d8b3ea9a2d4c74100d8b6ef16b6b1759a53bcab04ebb20a97bde/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:323 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/254ba5e2-5f70-4980-9f35-705260e6b4a8/volumes/kubernetes.io~projected/kube-api-access-fw56c DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d9b14ab60db68f4ac3bb5c36601a22207999a6d40df73fdb63600f06b1476cb/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:318 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/797ec6f1b718b17012edfeac42d352ade43b80fb1154187107c8d7eea975cfd2/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b48d395a1c66af09afa630d1d4c3b1173086b291478d651a40d154bc762f34c/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/640c3fc7954e29e1655ba46d9513b818a199820e185329f86fc11fadc3a5ebfb/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8efd9bc513a24549bd900282c046b98b288e016a1bcb15003ef484730cf3e589/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/299b108faa17dcf3a53dedb8a4ce351eb7ba8cbad39ad2c039d631c363be1e0a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~projected/kube-api-access-cbmv4 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~projected/kube-api-access-ddv27 DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0174417f-2cb0-4933-b9cc-7b044e2e8254/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:315 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:319 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d613c8d4-0ecb-4178-839a-e626bd2646e5/volumes/kubernetes.io~projected/kube-api-access-6vcnv DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/798e1f748fdb354fe892013fe551571574303c88599c06b8b117a3cbb6d91c0b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/476a01f5e0ff6e28b5400ad332b8a66d8de08001254210b49bf5673e72c7f7b0/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/cd31aaf2-4824-4357-818f-aaa8663c3917/volumes/kubernetes.io~projected/kube-api-access-nbfqf DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:321 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/487f4942-ea5d-44a2-92dc-9fed886c9fdf/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4fe728f134e0fdee62072b73b006a18cc2a5f5288b6f86df64dbd3dcdc765e7/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:244 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:323 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79e5efb821ab03a9dd347a84a09cf6686dadd40e44c445133bcc9d0217792f43/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:309 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5e735188-d000-4d3e-9e65-f3320352f3b5/volumes/kubernetes.io~projected/kube-api-access-jrz7t DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:320 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b96ecc89-8209-4f18-bc24-e96628147236/volumes/kubernetes.io~projected/kube-api-access-9ctqm DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f03d2523fff92b2dc272ef48c88c1a615f4990cbdab28aa4a8f34e39833cc96e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa3275d-c0c8-4ffb-9264-280b10d1672c/volumes/kubernetes.io~projected/kube-api-access-t9xzg DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6/volumes/kubernetes.io~projected/kube-api-access-n9jxn DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9acd3a44bfeb6da7231f277739838911133859045c1f02051a648081ac6b747a/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/316f1ae7-9bef-4691-a73d-e79f01fdea14/volumes/kubernetes.io~projected/kube-api-access-rwp9g DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac2fcf63-43d0-4759-9e77-edd3500ff388/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46e383e26addea4fc9566f042bf9f8e95e05d495fa7408e93df367a4ac593630/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7075e9e0-fca9-44ba-82bf-e93c702b576d/volumes/kubernetes.io~projected/kube-api-access-mlg5g DeviceMajor:0 DeviceMinor:357 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/487f4942-ea5d-44a2-92dc-9fed886c9fdf/volumes/kubernetes.io~projected/kube-api-access-g2cbk DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:361 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6/volumes/kubernetes.io~projected/kube-api-access-d6v8d DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d613c8d4-0ecb-4178-839a-e626bd2646e5/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/volumes/kubernetes.io~projected/kube-api-access-nqfgx DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~projected/kube-api-access-pl9br DeviceMajor:0 DeviceMinor:324 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/565f923e-a384-496e-be31-f99ddaec54ce/volumes/kubernetes.io~projected/kube-api-access-h4c6f DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be6522254cd3b3401be94428e8dad0893e20b4e599345b87ccef0f1c48e337fd/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40cc057e-3d16-464d-8b2c-7a15d01238c1/volumes/kubernetes.io~projected/kube-api-access-jstbl DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97e54fd23d3668217337c906798e1e4c6183940adf814521684ba405c97b189f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:322 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1eda2a1032d1a83c5918a766342e4640d6ec22344a790703393df474b308f048/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:323 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:323 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e47fab9093886b012e2a6436c400d52895195cf1906a2c544d8545c05cb5b91/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/254ba5e2-5f70-4980-9f35-705260e6b4a8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d22e902e725870996bbcbbfb764822114c7e343bda64364c9be8931c7ccf6ec0/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cd31aaf2-4824-4357-818f-aaa8663c3917/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:362 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1eda2a1032d1a83 MacAddress:1e:81:79:2d:86:fb Speed:10000 Mtu:8901} {Name:3e47fab9093886b MacAddress:2e:3f:80:10:5b:0c Speed:10000 Mtu:8901} {Name:476a01f5e0ff6e2 MacAddress:2a:0c:19:cc:7d:dc Speed:10000 Mtu:8901} {Name:797ec6f1b718b17 MacAddress:02:67:c9:39:05:19 Speed:10000 Mtu:8901} {Name:8efd9bc513a2454 MacAddress:7a:f7:ad:31:10:fe Speed:10000 Mtu:8901} {Name:9acd3a44bfeb6da MacAddress:36:9b:5b:54:8c:c9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d0:04:80:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:30:5a:9d:64:41 Speed:0 Mtu:8901} {Name:d22e902e7258709 MacAddress:76:bd:d7:f9:6c:69 Speed:10000 Mtu:8901} {Name:d4fe728f134e0fd MacAddress:42:dc:92:f6:6f:09 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d0:04:80:9f Speed:0 Mtu:9001} {Name:f03d2523fff92b2 MacAddress:d6:1c:0c:51:d6:3e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:86:fd:e0:39:c3:9b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:2f:a4:5f:85:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.851422    5817 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.851461    5817 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.852024    5817 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.852050    5817 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-233.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.852367    5817 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.852379    5817 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.852394    5817 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.852420    5817 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.852510    5817 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.852845    5817 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.853154    5817 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.853196    5817 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.853211    5817 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.853223    5817 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.853232    5817 kubelet.go:405] "Adding apiserver pod source"
Nov 03 10:29:29.853882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.853241    5817 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 10:29:29.854513 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.854059    5817 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 10:29:29.854513 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.854073    5817 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 10:29:29.854659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.854642    5817 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 10:29:29.856960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.856948    5817 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 10:29:29.856960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.856961    5817 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 10:29:29.857059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857050    5817 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 10:29:29.857231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857221    5817 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 10:29:29.857261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857250    5817 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 10:29:29.857343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857335    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 10:29:29.857375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857348    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 10:29:29.857375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857354    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 10:29:29.857375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857359    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 10:29:29.857375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857365    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 10:29:29.857375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857370    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 10:29:29.857375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857375    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 10:29:29.857532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857381    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 10:29:29.857532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857394    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 10:29:29.857532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857401    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 10:29:29.857532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857412    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 10:29:29.857532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857421    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 10:29:29.857532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857445    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 10:29:29.857532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.857451    5817 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 10:29:29.860692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.860680    5817 server.go:1263] "Started kubelet"
Nov 03 10:29:29.860751 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.860726    5817 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 10:29:29.860849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.860776    5817 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 10:29:29.860904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.860872    5817 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 10:29:29.861194 ip-10-0-138-233 systemd[1]: Started Kubernetes Kubelet.
Nov 03 10:29:29.861955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.861934    5817 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 10:29:29.863779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.863106    5817 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 10:29:29.866608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.866536    5817 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 10:29:29.867199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.867176    5817 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 10:29:29.868997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.867498    5817 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 10:29:29.868997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.867513    5817 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 10:29:29.868997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.867645    5817 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 10:29:29.870094 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.870048    5817 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:20 +0000 UTC" deadline="2028-05-02 17:12:04.352904618 +0000 UTC"
Nov 03 10:29:29.870094 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.870079    5817 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13110h42m34.482827783s"
Nov 03 10:29:29.870315 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:29.870284    5817 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found"
Nov 03 10:29:29.871118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871079    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fcd2d9d-803f-42cb-8db5-9f6a54c071c6" volumeName="kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 10:29:29.871216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871121    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fcd2d9d-803f-42cb-8db5-9f6a54c071c6" volumeName="kubernetes.io/secret/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 10:29:29.871216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871135    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0174417f-2cb0-4933-b9cc-7b044e2e8254" volumeName="kubernetes.io/configmap/0174417f-2cb0-4933-b9cc-7b044e2e8254-konnectivity-ca" seLinuxMountContext=""
Nov 03 10:29:29.871216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871150    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="487f4942-ea5d-44a2-92dc-9fed886c9fdf" volumeName="kubernetes.io/configmap/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 10:29:29.871216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871162    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 10:29:29.871216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871174    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 10:29:29.871216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871185    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 10:29:29.871216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871197    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config" seLinuxMountContext=""
Nov 03 10:29:29.871216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871211    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871222    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd" volumeName="kubernetes.io/projected/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-kube-api-access-cbmv4" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871235    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2" volumeName="kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871246    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31aaf2-4824-4357-818f-aaa8663c3917" volumeName="kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871260    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254ba5e2-5f70-4980-9f35-705260e6b4a8" volumeName="kubernetes.io/projected/254ba5e2-5f70-4980-9f35-705260e6b4a8-kube-api-access-fw56c" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871271    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40cc057e-3d16-464d-8b2c-7a15d01238c1" volumeName="kubernetes.io/projected/40cc057e-3d16-464d-8b2c-7a15d01238c1-kube-api-access-jstbl" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871286    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0174417f-2cb0-4933-b9cc-7b044e2e8254" volumeName="kubernetes.io/secret/0174417f-2cb0-4933-b9cc-7b044e2e8254-agent-certs" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871298    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40cc057e-3d16-464d-8b2c-7a15d01238c1" volumeName="kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-binary-copy" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871307    5817 factory.go:153] Registering CRI-O factory
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871309    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa3275d-c0c8-4ffb-9264-280b10d1672c" volumeName="kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871320    5817 factory.go:223] Registration of the crio container factory successfully
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871321    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac2fcf63-43d0-4759-9e77-edd3500ff388" volumeName="kubernetes.io/projected/ac2fcf63-43d0-4759-9e77-edd3500ff388-kube-api-access-ddv27" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871334    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa3275d-c0c8-4ffb-9264-280b10d1672c" volumeName="kubernetes.io/empty-dir/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-textfile" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871346    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2" volumeName="kubernetes.io/projected/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-api-access-nqfgx" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871359    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="487f4942-ea5d-44a2-92dc-9fed886c9fdf" volumeName="kubernetes.io/secret/487f4942-ea5d-44a2-92dc-9fed886c9fdf-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871370    5817 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871379    5817 factory.go:55] Registering systemd factory
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871386    5817 factory.go:223] Registration of the systemd container factory successfully
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871371    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fcd2d9d-803f-42cb-8db5-9f6a54c071c6" volumeName="kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-config" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871403    5817 factory.go:103] Registering Raw factory
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871407    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e" volumeName="kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871423    5817 manager.go:1199] Started watching for new ooms in manager
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871419    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2" volumeName="kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-metrics-client-ca" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871437    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2" volumeName="kubernetes.io/empty-dir/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-volume-directive-shadow" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871450    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="316f1ae7-9bef-4691-a73d-e79f01fdea14" volumeName="kubernetes.io/configmap/316f1ae7-9bef-4691-a73d-e79f01fdea14-iptables-alerter-script" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871462    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e735188-d000-4d3e-9e65-f3320352f3b5" volumeName="kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-daemon-config" seLinuxMountContext=""
Nov 03 10:29:29.871479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871473    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871492    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871504    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96ecc89-8209-4f18-bc24-e96628147236" volumeName="kubernetes.io/projected/b96ecc89-8209-4f18-bc24-e96628147236-kube-api-access-9ctqm" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871518    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="316f1ae7-9bef-4691-a73d-e79f01fdea14" volumeName="kubernetes.io/projected/316f1ae7-9bef-4691-a73d-e79f01fdea14-kube-api-access-rwp9g" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871531    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf53482-bc2a-4f95-a0ec-d2064baeb4f6" volumeName="kubernetes.io/projected/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-kube-api-access-n9jxn" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871543    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e735188-d000-4d3e-9e65-f3320352f3b5" volumeName="kubernetes.io/projected/5e735188-d000-4d3e-9e65-f3320352f3b5-kube-api-access-jrz7t" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871555    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa3275d-c0c8-4ffb-9264-280b10d1672c" volumeName="kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-metrics-client-ca" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871586    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871599    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa3275d-c0c8-4ffb-9264-280b10d1672c" volumeName="kubernetes.io/projected/6aa3275d-c0c8-4ffb-9264-280b10d1672c-kube-api-access-t9xzg" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871614    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac2fcf63-43d0-4759-9e77-edd3500ff388" volumeName="kubernetes.io/configmap/ac2fcf63-43d0-4759-9e77-edd3500ff388-metrics-client-ca" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871626    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565f923e-a384-496e-be31-f99ddaec54ce" volumeName="kubernetes.io/empty-dir/565f923e-a384-496e-be31-f99ddaec54ce-tmp-dir" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871639    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565f923e-a384-496e-be31-f99ddaec54ce" volumeName="kubernetes.io/projected/565f923e-a384-496e-be31-f99ddaec54ce-kube-api-access-h4c6f" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871652    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871667    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871680    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d613c8d4-0ecb-4178-839a-e626bd2646e5" volumeName="kubernetes.io/empty-dir/d613c8d4-0ecb-4178-839a-e626bd2646e5-tmp-dir" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871695    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254ba5e2-5f70-4980-9f35-705260e6b4a8" volumeName="kubernetes.io/secret/254ba5e2-5f70-4980-9f35-705260e6b4a8-cert" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871708    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="487f4942-ea5d-44a2-92dc-9fed886c9fdf" volumeName="kubernetes.io/projected/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-api-access-g2cbk" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871720    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa3275d-c0c8-4ffb-9264-280b10d1672c" volumeName="kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871733    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa3275d-c0c8-4ffb-9264-280b10d1672c" volumeName="kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-tls" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871746    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fcd2d9d-803f-42cb-8db5-9f6a54c071c6" volumeName="kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-env-overrides" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871759    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871771    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871783    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871795    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40cc057e-3d16-464d-8b2c-7a15d01238c1" volumeName="kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871807    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="487f4942-ea5d-44a2-92dc-9fed886c9fdf" volumeName="kubernetes.io/empty-dir/487f4942-ea5d-44a2-92dc-9fed886c9fdf-data-volume" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871819    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e735188-d000-4d3e-9e65-f3320352f3b5" volumeName="kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-cni-binary-copy" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871829    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871842    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd" volumeName="kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-tuned" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871852    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd31aaf2-4824-4357-818f-aaa8663c3917" volumeName="kubernetes.io/projected/cd31aaf2-4824-4357-818f-aaa8663c3917-kube-api-access-nbfqf" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871874    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d613c8d4-0ecb-4178-839a-e626bd2646e5" volumeName="kubernetes.io/configmap/d613c8d4-0ecb-4178-839a-e626bd2646e5-config-volume" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871885    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40cc057e-3d16-464d-8b2c-7a15d01238c1" volumeName="kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871895    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871906    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871927    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac2fcf63-43d0-4759-9e77-edd3500ff388" volumeName="kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871937    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d613c8d4-0ecb-4178-839a-e626bd2646e5" volumeName="kubernetes.io/projected/d613c8d4-0ecb-4178-839a-e626bd2646e5-kube-api-access-6vcnv" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871950    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd" volumeName="kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-tmp" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871962    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7075e9e0-fca9-44ba-82bf-e93c702b576d" volumeName="kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871972    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871982    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871995    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872007    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac2fcf63-43d0-4759-9e77-edd3500ff388" volumeName="kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872019    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2" volumeName="kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872030    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fcd2d9d-803f-42cb-8db5-9f6a54c071c6" volumeName="kubernetes.io/projected/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-kube-api-access-d6v8d" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872040    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d34fc88-a058-4770-b5e7-dea808286e2d" volumeName="kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872051    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2" volumeName="kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872063    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d613c8d4-0ecb-4178-839a-e626bd2646e5" volumeName="kubernetes.io/secret/d613c8d4-0ecb-4178-839a-e626bd2646e5-metrics-tls" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872074    5817 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf53482-bc2a-4f95-a0ec-d2064baeb4f6" volumeName="kubernetes.io/configmap/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-serviceca" seLinuxMountContext=""
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872084    5817 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872091    5817 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 10:29:29.872439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.871761    5817 manager.go:319] Starting recovery of all containers
Nov 03 10:29:29.874906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.872605    5817 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:29:29.876450 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.876433    5817 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:29:29.876779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.876761    5817 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:29:29.876872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.876830    5817 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 10:29:29.882319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.882294    5817 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 10:29:29.893212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.893188    5817 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 10:29:29.893312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.893215    5817 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 10:29:29.893312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.893249    5817 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 10:29:29.893312 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:29.893288    5817 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 10:29:29.896797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.896772    5817 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:29:29.906953 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.906928    5817 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="699d5f6b5dd51b089e1855624c49fe6a2c41ec0c23db679baee1b0e5f4b86595" exitCode=0
Nov 03 10:29:29.906953 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.906953    5817 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="c2cffd239a94fe5775cfc60026149e5932cbb85769e5af97a1f063711065c59d" exitCode=0
Nov 03 10:29:29.907059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.906959    5817 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="d8665006caa700bc799fd4b1b563539b41150751f29fc014d84a458d8de3571e" exitCode=0
Nov 03 10:29:29.907059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.906965    5817 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="3b365a1cccbe5c491b94ede9dafac81819aad4fdb62f9a5497629c99f86883e1" exitCode=0
Nov 03 10:29:29.907059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.906972    5817 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="23f8f0b8400e95c71f455e421077b7af835de3c653122302b157853df719169e" exitCode=0
Nov 03 10:29:29.907059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.906977    5817 generic.go:356] "Generic (PLEG): container finished" podID="40cc057e-3d16-464d-8b2c-7a15d01238c1" containerID="f8a2105165e83c189d7fcec4f77ee68376b9189f49170e2d77e54ce1cb4ec527" exitCode=0
Nov 03 10:29:29.926300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.926268    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0906d8be50774c4e8328161c8119d16" containerID="1204ee23ba3264cda244c0c385c03626b81ebb4306b3c411e559a8813163890e" exitCode=0
Nov 03 10:29:29.931429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.931408    5817 generic.go:356] "Generic (PLEG): container finished" podID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerID="8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d" exitCode=0
Nov 03 10:29:29.947303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.947271    5817 generic.go:356] "Generic (PLEG): container finished" podID="6aa3275d-c0c8-4ffb-9264-280b10d1672c" containerID="49036025d50bf7eb1f7300f2fc278bca315dccd3deaed28103e5debeda368e7c" exitCode=0
Nov 03 10:29:29.967345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.967329    5817 manager.go:324] Recovery completed
Nov 03 10:29:29.990467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990411    5817 cpu_manager.go:226] "Starting" policy="none"
Nov 03 10:29:29.990467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990427    5817 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 10:29:29.990467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990443    5817 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 10:29:29.990652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990612    5817 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 10:29:29.990652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990624    5817 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 10:29:29.990652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990642    5817 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 10:29:29.990652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990647    5817 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 10:29:29.990652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990653    5817 policy_none.go:50] "Start"
Nov 03 10:29:29.990786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990660    5817 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 10:29:29.990786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990670    5817 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 10:29:29.990838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990794    5817 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 10:29:29.990838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990801    5817 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 10:29:29.990838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.990806    5817 policy_none.go:44] "Start"
Nov 03 10:29:29.993378 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:29.993357    5817 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 10:29:29.995230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.995207    5817 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 10:29:29.997928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.997915    5817 manager.go:341] "Starting Device Plugin manager"
Nov 03 10:29:29.998477 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:29.997943    5817 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 10:29:29.998477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.998479    5817 server.go:87] "Starting device plugin registration server"
Nov 03 10:29:29.998751 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.998739    5817 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 10:29:29.998786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.998756    5817 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 10:29:29.998861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.998843    5817 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 10:29:29.998996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.998951    5817 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 10:29:29.999072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.999004    5817 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 10:29:29.999072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.999012    5817 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 10:29:29.999147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:29.999078    5817 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T10:29:29.998991502Z","UUID":"ce94b4a5-eb47-4452-8f0d-cfcd8b7e1000","Handler":null,"Name":"","Endpoint":""}
Nov 03 10:29:30.002408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.000675    5817 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 10:29:30.002408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.000711    5817 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 10:29:30.002530 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:30.002505    5817 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 10:29:30.099987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.099957    5817 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:29:30.103314 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.103296    5817 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:29:30.103431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.103326    5817 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:29:30.103431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.103337    5817 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:29:30.103431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.103375    5817 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.114550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.114532    5817 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.114676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.114604    5817 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.193817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.193776    5817 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"]
Nov 03 10:29:30.194337 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.194303    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"2cb181debb656c62bb70f1078166094d","Type":"ContainerStarted","Data":"776bcc34a786d8793385bfb78b24fe165a1bbcf1918f3ef2300bbcce41e05eb1"}
Nov 03 10:29:30.194431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.194345    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"2cb181debb656c62bb70f1078166094d","Type":"ContainerStarted","Data":"299b108faa17dcf3a53dedb8a4ce351eb7ba8cbad39ad2c039d631c363be1e0a"}
Nov 03 10:29:30.194431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.194354    5817 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.194431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.194363    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerStarted","Data":"5b11a1ef1302dbe6f933adcbc3efb545011d6add15a32b33cd160c821e136c79"}
Nov 03 10:29:30.194431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.194382    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerDied","Data":"1204ee23ba3264cda244c0c385c03626b81ebb4306b3c411e559a8813163890e"}
Nov 03 10:29:30.194431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.194400    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerStarted","Data":"46e383e26addea4fc9566f042bf9f8e95e05d495fa7408e93df367a4ac593630"}
Nov 03 10:29:30.194650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.194512    5817 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.205150 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.205137    5817 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:29:30.205187 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:30.205171    5817 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.207540 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.207526    5817 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:29:30.207598 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:30.207560    5817 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.278907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.278850    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.278907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.278879    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.278907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.278895    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.379451 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.379422    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.379603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.379456    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.379603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.379481    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.379603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.379540    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.379603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.379540    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.379749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.379545    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.853798 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.853756    5817 apiserver.go:52] "Watching apiserver"
Nov 03 10:29:30.861209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.861182    5817 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 10:29:30.861738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.861718    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zwj2x","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-pf78w","openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz","openshift-image-registry/node-ca-cxvr7","openshift-monitoring/node-exporter-mpkfs","openshift-ovn-kubernetes/ovnkube-node-j6sfm","openshift-dns/node-resolver-9bmpc","openshift-multus/network-metrics-daemon-4qhrr","kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-d9x57","openshift-dns/dns-default-wtrwm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-6kr4v","openshift-cluster-node-tuning-operator/tuned-fqldz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n","openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68","openshift-multus/multus-additional-cni-plugins-jtzkm","kube-system/konnectivity-agent-8v4lr","openshift-insights/insights-runtime-extractor-jq8jl","openshift-network-diagnostics/network-check-target-brkhs"]
Nov 03 10:29:30.866662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.866547    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xjpbn\""
Nov 03 10:29:30.867304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.867249    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 10:29:30.868021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.867994    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.868128 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.868119    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.868240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.868221    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ssvsh\""
Nov 03 10:29:30.868395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.868375    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 10:29:30.868644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.868620    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nxhw\""
Nov 03 10:29:30.868747 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.868228    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.868874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.868857    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.869050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.869033    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 10:29:30.869196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.869178    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.869352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.869340    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 10:29:30.869705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.869687    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 10:29:30.869859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.869420    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.869943 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.869926    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.869998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.869959    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 10:29:30.870068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.869783    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.870210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870058    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.870483 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870056    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.870610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870083    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 10:29:30.870698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870210    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 10:29:30.870834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870815    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wbpkj\""
Nov 03 10:29:30.870907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870324    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.870907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870391    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 10:29:30.870907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870516    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-trm8g\""
Nov 03 10:29:30.870907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870611    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.871101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870906    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.871101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870671    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.871101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.871038    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pthxs\""
Nov 03 10:29:30.871101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870229    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.871101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.870741    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4c7q2\""
Nov 03 10:29:30.871588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.871552    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 10:29:30.871588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.871560    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 10:29:30.871588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.871562    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zcr5r\""
Nov 03 10:29:30.871703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.871552    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jkrmv\""
Nov 03 10:29:30.872079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.872063    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.872163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.872147    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 10:29:30.872230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.872212    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 10:29:30.872308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.872260    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5hs5j\""
Nov 03 10:29:30.872460 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.872431    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 10:29:30.872460 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.872432    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-khtn4\""
Nov 03 10:29:30.872604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.872482    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 10:29:30.872676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.872662    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lq8vs\""
Nov 03 10:29:30.873789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.873769    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.874031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.874015    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w9nrb\""
Nov 03 10:29:30.874752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.874695    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.874845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.874818    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w9j8\""
Nov 03 10:29:30.875092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875076    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 10:29:30.875243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875217    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 10:29:30.875307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875274    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 10:29:30.875351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875332    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 10:29:30.875676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:30.875676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875660    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 10:29:30.875814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875667    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.875866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875851    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 10:29:30.875923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.875875    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.876201 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.876188    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 10:29:30.876333 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.876321    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 10:29:30.876865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.876843    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 10:29:30.876995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.876867    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 10:29:30.876995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.876965    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 10:29:30.877092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877078    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjjb4\""
Nov 03 10:29:30.877166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877116    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 10:29:30.877166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877151    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 10:29:30.877279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877176    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 10:29:30.877279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877197    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 10:29:30.877279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877221    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 10:29:30.877279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877266    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-szl4w\""
Nov 03 10:29:30.877387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877270    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gzgf5\""
Nov 03 10:29:30.877422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.877386    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5cc62\""
Nov 03 10:29:30.878129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878112    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 10:29:30.878179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878128    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 10:29:30.878179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878131    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 10:29:30.878179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878161    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 10:29:30.878179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878173    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 10:29:30.878378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878208    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cb568\""
Nov 03 10:29:30.878378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878248    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 10:29:30.878378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878250    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 10:29:30.878378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878268    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d9v2v0thtuv8v\""
Nov 03 10:29:30.878550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878536    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 10:29:30.878550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878548    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 10:29:30.878834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878819    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 10:29:30.878944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878931    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 10:29:30.878998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878936    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 10:29:30.878998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.878948    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 10:29:30.881657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881637    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-sys\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.881737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881671    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-lib-modules\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.881737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881697    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-bin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.881737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881721    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-etc-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.881841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881795    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-netd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.881841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881827    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-script-lib\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.881926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881853    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-system-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.881926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881886    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.882025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881925    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-multus-certs\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.882025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881956    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v8d\" (UniqueName: \"kubernetes.io/projected/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-kube-api-access-d6v8d\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.882025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881981    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/316f1ae7-9bef-4691-a73d-e79f01fdea14-iptables-alerter-script\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:29:30.882025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.881997    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/316f1ae7-9bef-4691-a73d-e79f01fdea14-host-slash\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:29:30.882210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882029    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-serviceca\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:29:30.882210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882048    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:30.882210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882081    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-tmp\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.882210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882104    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-bin\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.882210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882119    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-cnibin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.882210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882103    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-script-lib\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.882210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882198    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/316f1ae7-9bef-4691-a73d-e79f01fdea14-iptables-alerter-script\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882214    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-host\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882246    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-cnibin\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882275    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882333    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd31aaf2-4824-4357-818f-aaa8663c3917-metrics-certs\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882362    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882372    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-tmp\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882450    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882487    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882489    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-kubelet\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882491    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.882532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882532    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-kubelet-config\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882559    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-host\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882605    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-cni-binary-copy\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882659    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-socket-dir-parent\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882697    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-netns\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882718    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882731    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw56c\" (UniqueName: \"kubernetes.io/projected/254ba5e2-5f70-4980-9f35-705260e6b4a8-kube-api-access-fw56c\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882759    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882788    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-sys-fs\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882790    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-serviceca\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882856    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-cni-binary-copy\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882858    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrz7t\" (UniqueName: \"kubernetes.io/projected/5e735188-d000-4d3e-9e65-f3320352f3b5-kube-api-access-jrz7t\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882894    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfqf\" (UniqueName: \"kubernetes.io/projected/cd31aaf2-4824-4357-818f-aaa8663c3917-kube-api-access-nbfqf\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:30.883008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.882952    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40cc057e-3d16-464d-8b2c-7a15d01238c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883030    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-ovn\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883092    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-node-log\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883118    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-systemd-units\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883149    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-slash\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883172    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883194    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-var-lib-kubelet\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883213    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/565f923e-a384-496e-be31-f99ddaec54ce-hosts-file\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883247    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883272    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-dbus\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883336    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctqm\" (UniqueName: \"kubernetes.io/projected/b96ecc89-8209-4f18-bc24-e96628147236-kube-api-access-9ctqm\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883406    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-tuned\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883506    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-tuned\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.883544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883484    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-conf-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883554    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwp9g\" (UniqueName: \"kubernetes.io/projected/316f1ae7-9bef-4691-a73d-e79f01fdea14-kube-api-access-rwp9g\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883625    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-run\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883657    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmv4\" (UniqueName: \"kubernetes.io/projected/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-kube-api-access-cbmv4\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883703    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883818    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jxn\" (UniqueName: \"kubernetes.io/projected/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-kube-api-access-n9jxn\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883856    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-daemon-config\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883881    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254ba5e2-5f70-4980-9f35-705260e6b4a8-cert\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883931    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-kubernetes\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883951    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.884066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.883985    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-kubelet\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884072    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-daemon-config\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884082    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884110    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-socket-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884150    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254ba5e2-5f70-4980-9f35-705260e6b4a8-cert\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884172    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-registration-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884218    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-modprobe-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884255    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-etc-kubernetes\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884278    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-systemd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884324    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-conf\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884347    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-k8s-cni-cncf-io\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884363    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-original-pull-secret\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884368    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565f923e-a384-496e-be31-f99ddaec54ce-tmp-dir\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884416    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-log-socket\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.884452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884423    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565f923e-a384-496e-be31-f99ddaec54ce-tmp-dir\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884468    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-var-lib-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884510    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884528    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0174417f-2cb0-4933-b9cc-7b044e2e8254-agent-certs\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884566    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4c6f\" (UniqueName: \"kubernetes.io/projected/565f923e-a384-496e-be31-f99ddaec54ce-kube-api-access-h4c6f\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884626    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884653    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-env-overrides\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884687    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstbl\" (UniqueName: \"kubernetes.io/projected/40cc057e-3d16-464d-8b2c-7a15d01238c1-kube-api-access-jstbl\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884773    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0174417f-2cb0-4933-b9cc-7b044e2e8254-agent-certs\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884808    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-env-overrides\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884828    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884874    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysconfig\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884903    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-systemd\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884933    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0174417f-2cb0-4933-b9cc-7b044e2e8254-konnectivity-ca\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.884959    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-multus\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885008    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-config\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885034    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovn-node-metrics-cert\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885084    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0174417f-2cb0-4933-b9cc-7b044e2e8254-konnectivity-ca\") pod \"konnectivity-agent-8v4lr\" (UID: \"0174417f-2cb0-4933-b9cc-7b044e2e8254\") " pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885113    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-netns\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885140    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-device-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885162    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-hostroot\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.885163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885171    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovnkube-config\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.885765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885202    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.885765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885243    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.885765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.885288    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-ovn-node-metrics-cert\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.888214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.888197    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v8d\" (UniqueName: \"kubernetes.io/projected/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-kube-api-access-d6v8d\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.888901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.888884    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctqm\" (UniqueName: \"kubernetes.io/projected/b96ecc89-8209-4f18-bc24-e96628147236-kube-api-access-9ctqm\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.888942 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.888909    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw56c\" (UniqueName: \"kubernetes.io/projected/254ba5e2-5f70-4980-9f35-705260e6b4a8-kube-api-access-fw56c\") pod \"ingress-canary-d9x57\" (UID: \"254ba5e2-5f70-4980-9f35-705260e6b4a8\") " pod="openshift-ingress-canary/ingress-canary-d9x57"
Nov 03 10:29:30.889519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.889427    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrz7t\" (UniqueName: \"kubernetes.io/projected/5e735188-d000-4d3e-9e65-f3320352f3b5-kube-api-access-jrz7t\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.889626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.889598    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfqf\" (UniqueName: \"kubernetes.io/projected/cd31aaf2-4824-4357-818f-aaa8663c3917-kube-api-access-nbfqf\") pod \"network-metrics-daemon-4qhrr\" (UID: \"cd31aaf2-4824-4357-818f-aaa8663c3917\") " pod="openshift-multus/network-metrics-daemon-4qhrr"
Nov 03 10:29:30.890313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.889765    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmv4\" (UniqueName: \"kubernetes.io/projected/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-kube-api-access-cbmv4\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.890313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.890097    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jxn\" (UniqueName: \"kubernetes.io/projected/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-kube-api-access-n9jxn\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:29:30.890313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.890149    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwp9g\" (UniqueName: \"kubernetes.io/projected/316f1ae7-9bef-4691-a73d-e79f01fdea14-kube-api-access-rwp9g\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:29:30.892628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.891071    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlg5g\" (UniqueName: \"kubernetes.io/projected/7075e9e0-fca9-44ba-82bf-e93c702b576d-kube-api-access-mlg5g\") pod \"network-check-target-brkhs\" (UID: \"7075e9e0-fca9-44ba-82bf-e93c702b576d\") " pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:30.892628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.891120    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstbl\" (UniqueName: \"kubernetes.io/projected/40cc057e-3d16-464d-8b2c-7a15d01238c1-kube-api-access-jstbl\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.892628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.891537    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4c6f\" (UniqueName: \"kubernetes.io/projected/565f923e-a384-496e-be31-f99ddaec54ce-kube-api-access-h4c6f\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:29:30.954143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.954122    5817 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.954272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.954260    5817 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.965795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.965771    5817 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:29:30.965926 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:30.965824    5817 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.966034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.965957    5817 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:29:30.966638 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:30.966615    5817 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"
Nov 03 10:29:30.969136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.969108    5817 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 10:29:30.986088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986063    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.986196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986090    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac2fcf63-43d0-4759-9e77-edd3500ff388-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:30.986196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986107    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.986196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986130    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-dbus\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:30.986196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986182    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986193    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986187    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-dbus\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986222    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-conf-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986255    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986231    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-conf-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986283    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-run\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986304    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d613c8d4-0ecb-4178-839a-e626bd2646e5-metrics-tls\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986321    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986327    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-run\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986319    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac2fcf63-43d0-4759-9e77-edd3500ff388-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986343    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-sys\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.986392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986376    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986459    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986473    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986501    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d613c8d4-0ecb-4178-839a-e626bd2646e5-metrics-tls\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986503    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986540    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-kubelet\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986558    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-socket-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986602    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-registration-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986612    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-kubelet\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986640    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986640    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986656    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-registration-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986620    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-modprobe-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986689    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-socket-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986693    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986701    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-modprobe-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986741    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986811    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-k8s-cni-cncf-io\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986901    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-log-socket\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986906    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986929    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-k8s-cni-cncf-io\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986930    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-tls\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986945    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986951    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-log-socket\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986969    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.986993    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987012    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.987030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987031    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysconfig\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987063    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987078    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysconfig\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987101    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987117    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-tls\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987122    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/487f4942-ea5d-44a2-92dc-9fed886c9fdf-crio-socket\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987144    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987165    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987174    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-netns\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987201    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987225    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-device-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987242    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987269    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-run-netns\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987283    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-device-dir\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987304    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987356    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-lib-modules\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987378    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987415    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-d\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987448    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-lib-modules\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987459    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987519    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-etc-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987535    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-netd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987550    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-system-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987556    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-netd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987562    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987535    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-etc-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987565    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-multus-certs\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987610    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-multus-certs\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987616    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/316f1ae7-9bef-4691-a73d-e79f01fdea14-host-slash\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987628    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-system-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987633    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-textfile\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987660    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-host\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987680    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d613c8d4-0ecb-4178-839a-e626bd2646e5-config-volume\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987685    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-textfile\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987692    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bf53482-bc2a-4f95-a0ec-d2064baeb4f6-host\") pod \"node-ca-cxvr7\" (UID: \"3bf53482-bc2a-4f95-a0ec-d2064baeb4f6\") " pod="openshift-image-registry/node-ca-cxvr7"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987700    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/316f1ae7-9bef-4691-a73d-e79f01fdea14-host-slash\") pod \"iptables-alerter-6kr4v\" (UID: \"316f1ae7-9bef-4691-a73d-e79f01fdea14\") " pod="openshift-network-operator/iptables-alerter-6kr4v"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987706    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987734    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/487f4942-ea5d-44a2-92dc-9fed886c9fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987756    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-socket-dir-parent\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987781    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-netns\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987809    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-root\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987834    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987855    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-socket-dir-parent\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987862    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d613c8d4-0ecb-4178-839a-e626bd2646e5-config-volume\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987855    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987858    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987921    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-sys-fs\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987944    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-run-netns\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987948    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987966    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.987986    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988005    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-sys-fs\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988013    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-ovn\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988027    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-node-log\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988031    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988063    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-ovn\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988067    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-systemd-units\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988073    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/487f4942-ea5d-44a2-92dc-9fed886c9fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988090    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-node-log\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988093    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-systemd-units\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988097    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988115    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-slash\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988233    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-var-lib-kubelet\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988244    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-slash\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988286    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988294    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/565f923e-a384-496e-be31-f99ddaec54ce-hosts-file\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:29:30.988307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988330    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/565f923e-a384-496e-be31-f99ddaec54ce-hosts-file\") pod \"node-resolver-9bmpc\" (UID: \"565f923e-a384-496e-be31-f99ddaec54ce\") " pod="openshift-dns/node-resolver-9bmpc"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988373    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988408    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988425    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-var-lib-kubelet\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988466    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988602    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-wtmp\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988621    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988631    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988660    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-kubernetes\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988682    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-etc-kubernetes\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988703    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988784    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-etc-kubernetes\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988785    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988789    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988821    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-kubernetes\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988834    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac2fcf63-43d0-4759-9e77-edd3500ff388-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988867    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988906    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-systemd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988922    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-conf\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988949    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988979    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-run-systemd\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.988994    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989061    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-sysctl-conf\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989110    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989113    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989129    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9br\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989146    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-var-lib-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989158    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989169    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989196    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddv27\" (UniqueName: \"kubernetes.io/projected/ac2fcf63-43d0-4759-9e77-edd3500ff388-kube-api-access-ddv27\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989218    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989207    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-var-lib-openvswitch\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989241    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-systemd\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989254    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989264    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-multus\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989287    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-hostroot\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989306    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-etc-systemd\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989345    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989346    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-multus\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989309    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcnv\" (UniqueName: \"kubernetes.io/projected/d613c8d4-0ecb-4178-839a-e626bd2646e5-kube-api-access-6vcnv\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989307    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96ecc89-8209-4f18-bc24-e96628147236-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkc7n\" (UID: \"b96ecc89-8209-4f18-bc24-e96628147236\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkc7n"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989363    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-hostroot\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989403    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989429    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989455    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/487f4942-ea5d-44a2-92dc-9fed886c9fdf-data-volume\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989477    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989499    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-sys\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989522    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-bin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989555    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/487f4942-ea5d-44a2-92dc-9fed886c9fdf-data-volume\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989589    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989592    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989609    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-sys\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989615    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-cni-bin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989615    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989661    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d613c8d4-0ecb-4178-839a-e626bd2646e5-tmp-dir\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989663    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-multus-cni-dir\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989686    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqfgx\" (UniqueName: \"kubernetes.io/projected/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-api-access-nqfgx\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989705    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989714    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d613c8d4-0ecb-4178-839a-e626bd2646e5-tmp-dir\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989720    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-bin\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989740    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-cnibin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989763    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-cnibin\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989789    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-kubelet\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989797    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989817    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-metrics-client-ca\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989844    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989866    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-cnibin\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989868    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-kubelet-config\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989901    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40cc057e-3d16-464d-8b2c-7a15d01238c1-cnibin\") pod \"multus-additional-cni-plugins-jtzkm\" (UID: \"40cc057e-3d16-464d-8b2c-7a15d01238c1\") " pod="openshift-multus/multus-additional-cni-plugins-jtzkm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989904    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-host\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989927    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e-kubelet-config\") pod \"global-pull-secret-syncer-zwj2x\" (UID: \"8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e\") " pod="kube-system/global-pull-secret-syncer-zwj2x"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989936    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcd2d9d-803f-42cb-8db5-9f6a54c071c6-host-cni-bin\") pod \"ovnkube-node-j6sfm\" (UID: \"6fcd2d9d-803f-42cb-8db5-9f6a54c071c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989879    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e735188-d000-4d3e-9e65-f3320352f3b5-host-var-lib-kubelet\") pod \"multus-pf78w\" (UID: \"5e735188-d000-4d3e-9e65-f3320352f3b5\") " pod="openshift-multus/multus-pf78w"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989935    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.989937    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.990059    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd-host\") pod \"tuned-fqldz\" (UID: \"ab1e2b76-dfe9-46d8-a82e-b837bbab1bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-fqldz"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.990066    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.990084    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cbk\" (UniqueName: \"kubernetes.io/projected/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-api-access-g2cbk\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.990100    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.990116    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xzg\" (UniqueName: \"kubernetes.io/projected/6aa3275d-c0c8-4ffb-9264-280b10d1672c-kube-api-access-t9xzg\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.990176    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3275d-c0c8-4ffb-9264-280b10d1672c-metrics-client-ca\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.990533    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.991205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.990566    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.999503 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.999477    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9br\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br\") pod \"prometheus-k8s-0\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:30.999692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.999512    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddv27\" (UniqueName: \"kubernetes.io/projected/ac2fcf63-43d0-4759-9e77-edd3500ff388-kube-api-access-ddv27\") pod \"openshift-state-metrics-7bd7489957-8lfdz\" (UID: \"ac2fcf63-43d0-4759-9e77-edd3500ff388\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8lfdz"
Nov 03 10:29:30.999887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:30.999868    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cbk\" (UniqueName: \"kubernetes.io/projected/487f4942-ea5d-44a2-92dc-9fed886c9fdf-kube-api-access-g2cbk\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:31.000066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.000047    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcnv\" (UniqueName: \"kubernetes.io/projected/d613c8d4-0ecb-4178-839a-e626bd2646e5-kube-api-access-6vcnv\") pod \"dns-default-wtrwm\" (UID: \"d613c8d4-0ecb-4178-839a-e626bd2646e5\") " pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:31.000253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.000094    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xzg\" (UniqueName: \"kubernetes.io/projected/6aa3275d-c0c8-4ffb-9264-280b10d1672c-kube-api-access-t9xzg\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:31.000253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.000162    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqfgx\" (UniqueName: \"kubernetes.io/projected/bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2-kube-api-access-nqfgx\") pod \"kube-state-metrics-5dc8d7d89b-xqq68\" (UID: \"bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xqq68"
Nov 03 10:29:31.090788 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.090762    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/487f4942-ea5d-44a2-92dc-9fed886c9fdf-crio-socket\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:31.090878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.090797    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-root\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:31.090878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.090820    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-wtmp\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:31.090878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.090870    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-sys\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:31.090977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.090876    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/487f4942-ea5d-44a2-92dc-9fed886c9fdf-crio-socket\") pod \"insights-runtime-extractor-jq8jl\" (UID: \"487f4942-ea5d-44a2-92dc-9fed886c9fdf\") " pod="openshift-insights/insights-runtime-extractor-jq8jl"
Nov 03 10:29:31.090977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.090879    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-root\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:31.090977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.090919    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-sys\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:31.091056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:31.090973    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6aa3275d-c0c8-4ffb-9264-280b10d1672c-node-exporter-wtmp\") pod \"node-exporter-mpkfs\" (UID: \"6aa3275d-c0c8-4ffb-9264-280b10d1672c\") " pod="openshift-monitoring/node-exporter-mpkfs"
Nov 03 10:29:32.367356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:32.367326    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:33.283214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:33.283178    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:34.047895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:34.047863    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wtrwm"
Nov 03 10:29:34.394014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:34.393937    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:34.520334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:34.520304    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:36.017552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:36.017524    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:36.017925 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:36.017733    5817 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 10:29:38.200342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:38.200311    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:29:41.993481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:41.993405    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:42.181451 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:42.181415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:44.505808 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:44.505775    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8v4lr"
Nov 03 10:29:44.511383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:44.511357    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:44.636636 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:44.636610    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:49.842180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:49.842146    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:49.842718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:49.842673    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy" containerID="cri-o://d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8" gracePeriod=600
Nov 03 10:29:49.842718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:49.842682    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="thanos-sidecar" containerID="cri-o://009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d" gracePeriod=600
Nov 03 10:29:49.842888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:49.842716    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="config-reloader" containerID="cri-o://46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6" gracePeriod=600
Nov 03 10:29:49.842888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:49.842658    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="prometheus" containerID="cri-o://222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474" gracePeriod=600
Nov 03 10:29:49.842888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:49.842716    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy-web" containerID="cri-o://f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6" gracePeriod=600
Nov 03 10:29:49.842888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:49.842775    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae" gracePeriod=600
Nov 03 10:29:49.882041 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:49.882018    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6sfm"
Nov 03 10:29:50.016730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016700    5817 generic.go:356] "Generic (PLEG): container finished" podID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerID="d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8" exitCode=0
Nov 03 10:29:50.016730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016733    5817 generic.go:356] "Generic (PLEG): container finished" podID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerID="f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6" exitCode=0
Nov 03 10:29:50.016910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016746    5817 generic.go:356] "Generic (PLEG): container finished" podID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerID="009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d" exitCode=0
Nov 03 10:29:50.016910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016753    5817 generic.go:356] "Generic (PLEG): container finished" podID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerID="46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6" exitCode=0
Nov 03 10:29:50.016910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016758    5817 generic.go:356] "Generic (PLEG): container finished" podID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerID="222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474" exitCode=0
Nov 03 10:29:50.016910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016778    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerDied","Data":"d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8"}
Nov 03 10:29:50.016910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016853    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerDied","Data":"f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6"}
Nov 03 10:29:50.016910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016882    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerDied","Data":"009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d"}
Nov 03 10:29:50.016910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016907    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerDied","Data":"46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6"}
Nov 03 10:29:50.017117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.016936    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerDied","Data":"222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474"}
Nov 03 10:29:50.219137 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.219105    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:50.316537 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316509    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.316724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316549    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.316724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316584    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.316724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316606    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.316724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316632    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316768    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316828    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316862    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316891    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316934    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.316968    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317003    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317049    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317085    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317118    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317146    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317209    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317377    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.317397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317404    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.318155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317441    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs\") pod \"9d34fc88-a058-4770-b5e7-dea808286e2d\" (UID: \"9d34fc88-a058-4770-b5e7-dea808286e2d\") "
Nov 03 10:29:50.318155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317620    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:50.318155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.317740    5817 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.319820 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.318926    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:29:50.319820 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.319452    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:50.319820 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.319509    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:50.320053 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.320000    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "kube-api-access-pl9br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:29:50.320116 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.320089    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:50.320116 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.320097    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:50.320803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.320392    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.320803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.320461    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:50.320803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.320533    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:50.320803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.320726    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.321186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.321163    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:29:50.321694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.321655    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.322113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.322089    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.322113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.322102    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:29:50.322267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.322242    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.322393 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.322375    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.322454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.322434    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.322773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.322756    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.330948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.330929    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config" pod "9d34fc88-a058-4770-b5e7-dea808286e2d" (UID: "9d34fc88-a058-4770-b5e7-dea808286e2d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:50.418794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418739    5817 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.418794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418762    5817 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-config-out\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.418794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418772    5817 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.418794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418781    5817 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-grpc-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.418794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418791    5817 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-configmap-metrics-client-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418801    5817 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418809    5817 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418829    5817 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-metrics-client-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418837    5817 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-kube-rbac-proxy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418845    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl9br\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-kube-api-access-pl9br\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418853    5817 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d34fc88-a058-4770-b5e7-dea808286e2d-tls-assets\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418861    5817 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-db\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418870    5817 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-config\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418879    5817 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-web-config\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418887    5817 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d34fc88-a058-4770-b5e7-dea808286e2d-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418896    5817 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418904    5817 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418912    5817 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:50.419001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:50.418920    5817 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d34fc88-a058-4770-b5e7-dea808286e2d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:51.023923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.023890    5817 generic.go:356] "Generic (PLEG): container finished" podID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerID="e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae" exitCode=0
Nov 03 10:29:51.024306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.023954    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerDied","Data":"e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae"}
Nov 03 10:29:51.024306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.023992    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.024306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.024025    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d34fc88-a058-4770-b5e7-dea808286e2d","Type":"ContainerDied","Data":"1eda2a1032d1a83c5918a766342e4640d6ec22344a790703393df474b308f048"}
Nov 03 10:29:51.024306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.024075    5817 scope.go:122] "RemoveContainer" containerID="e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae"
Nov 03 10:29:51.034344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.034330    5817 scope.go:122] "RemoveContainer" containerID="d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8"
Nov 03 10:29:51.041608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.041589    5817 scope.go:122] "RemoveContainer" containerID="f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6"
Nov 03 10:29:51.048018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.048001    5817 scope.go:122] "RemoveContainer" containerID="009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d"
Nov 03 10:29:51.049937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.049917    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:51.054170 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.054146    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:51.055557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.055540    5817 scope.go:122] "RemoveContainer" containerID="46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6"
Nov 03 10:29:51.062619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.062600    5817 scope.go:122] "RemoveContainer" containerID="222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474"
Nov 03 10:29:51.069252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.069234    5817 scope.go:122] "RemoveContainer" containerID="8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d"
Nov 03 10:29:51.075645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.075630    5817 scope.go:122] "RemoveContainer" containerID="e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae"
Nov 03 10:29:51.075916 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:51.075898    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae\": container with ID starting with e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae not found: ID does not exist" containerID="e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae"
Nov 03 10:29:51.075976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.075929    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae"} err="failed to get container status \"e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae\": rpc error: code = NotFound desc = could not find container \"e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae\": container with ID starting with e7507359820a0b85cf10c28896e0044986f03545f482f37d08a7d74d0046bbae not found: ID does not exist"
Nov 03 10:29:51.075976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.075972    5817 scope.go:122] "RemoveContainer" containerID="d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8"
Nov 03 10:29:51.076222 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:51.076205    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8\": container with ID starting with d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8 not found: ID does not exist" containerID="d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8"
Nov 03 10:29:51.076278 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.076254    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8"} err="failed to get container status \"d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8\": rpc error: code = NotFound desc = could not find container \"d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8\": container with ID starting with d26301901c07ede633fc84e862b9227cff47239cd55a4ee531c58ed63852d2a8 not found: ID does not exist"
Nov 03 10:29:51.076278 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.076272    5817 scope.go:122] "RemoveContainer" containerID="f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6"
Nov 03 10:29:51.076481 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:51.076465    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6\": container with ID starting with f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6 not found: ID does not exist" containerID="f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6"
Nov 03 10:29:51.076525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.076485    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6"} err="failed to get container status \"f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6\": rpc error: code = NotFound desc = could not find container \"f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6\": container with ID starting with f0c0403fd675d330fb270722812897b23e193e5f5bd0c49e542e5d81836e45c6 not found: ID does not exist"
Nov 03 10:29:51.076525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.076497    5817 scope.go:122] "RemoveContainer" containerID="009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d"
Nov 03 10:29:51.076736 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:51.076720    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d\": container with ID starting with 009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d not found: ID does not exist" containerID="009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d"
Nov 03 10:29:51.076781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.076739    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d"} err="failed to get container status \"009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d\": rpc error: code = NotFound desc = could not find container \"009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d\": container with ID starting with 009a6cbcb267ebe09a2e1ad0862139f68f8b306605d5341ba5806988e53a697d not found: ID does not exist"
Nov 03 10:29:51.076781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.076751    5817 scope.go:122] "RemoveContainer" containerID="46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6"
Nov 03 10:29:51.076957 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:51.076944    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6\": container with ID starting with 46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6 not found: ID does not exist" containerID="46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6"
Nov 03 10:29:51.076995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.076962    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6"} err="failed to get container status \"46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6\": rpc error: code = NotFound desc = could not find container \"46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6\": container with ID starting with 46fdce37101b9ecddde8178277d75ff6690f6369fee318d925121fbea19321a6 not found: ID does not exist"
Nov 03 10:29:51.076995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.076973    5817 scope.go:122] "RemoveContainer" containerID="222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474"
Nov 03 10:29:51.077153 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:51.077139    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474\": container with ID starting with 222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474 not found: ID does not exist" containerID="222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474"
Nov 03 10:29:51.077193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.077156    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474"} err="failed to get container status \"222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474\": rpc error: code = NotFound desc = could not find container \"222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474\": container with ID starting with 222188780a17f72ede53971d4d5b10f2f72101cd6fda1d1a3d260a0744677474 not found: ID does not exist"
Nov 03 10:29:51.077193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.077167    5817 scope.go:122] "RemoveContainer" containerID="8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d"
Nov 03 10:29:51.077344 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:51.077330    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d\": container with ID starting with 8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d not found: ID does not exist" containerID="8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d"
Nov 03 10:29:51.077381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.077346    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d"} err="failed to get container status \"8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d\": rpc error: code = NotFound desc = could not find container \"8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d\": container with ID starting with 8a64020280f89395ea5d3007a13a31ca265235b7dd4d58e73c0250c09565473d not found: ID does not exist"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079672    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079912    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="thanos-sidecar"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079927    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="thanos-sidecar"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079950    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079955    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079968    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy-thanos"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079973    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy-thanos"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079983    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy-web"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079987    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy-web"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.079998    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="config-reloader"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080005    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="config-reloader"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080016    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="prometheus"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080024    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="prometheus"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080029    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="init-config-reloader"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080035    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="init-config-reloader"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080076    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080084    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy-thanos"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080092    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="kube-rbac-proxy-web"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080100    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="config-reloader"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080105    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="prometheus"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080110    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="init-config-reloader"
Nov 03 10:29:51.080607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.080115    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" containerName="thanos-sidecar"
Nov 03 10:29:51.115876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.115850    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:51.115989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.115974    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.120687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120634    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 10:29:51.120687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120667    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 10:29:51.120687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120676    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 10:29:51.120687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120682    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 10:29:51.120954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120682    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 10:29:51.120954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120706    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 10:29:51.121056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120935    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 10:29:51.121056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120981    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 10:29:51.121056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.120954    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 10:29:51.121208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.121118    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 10:29:51.121208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.121133    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cb568\""
Nov 03 10:29:51.121208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.121143    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 10:29:51.121208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.121193    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 10:29:51.121391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.121281    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 10:29:51.122232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.122213    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d9v2v0thtuv8v\""
Nov 03 10:29:51.125768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.125753    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 10:29:51.125869 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.125854    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 10:29:51.222816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.222795    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-config\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.222921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.222822    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.222921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.222843    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.222985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.222915    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.222985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.222947    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223011    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223032    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223053    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223076    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223132    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223186    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223207    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223222    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgjq\" (UniqueName: \"kubernetes.io/projected/ce5df295-143c-4805-aafc-5f700c804cab-kube-api-access-txgjq\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223266    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce5df295-143c-4805-aafc-5f700c804cab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223290    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223321    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223355    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223373    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223394    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.223480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.223439    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce5df295-143c-4805-aafc-5f700c804cab-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324085    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324110    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324127    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324143    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324165    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324195    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324221    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324248    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324271    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324290    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgjq\" (UniqueName: \"kubernetes.io/projected/ce5df295-143c-4805-aafc-5f700c804cab-kube-api-access-txgjq\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324317    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce5df295-143c-4805-aafc-5f700c804cab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324339    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324385    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324416    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324438    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324473    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324502    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce5df295-143c-4805-aafc-5f700c804cab-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324530    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-config\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324562    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324596    5817 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 10:29:51.324842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324616    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.324842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324808    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.325230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324895    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.325230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.324968    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.325886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.325862    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.327293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.327268    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce5df295-143c-4805-aafc-5f700c804cab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.328239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.327603    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.328239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.327671    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.328239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.327671    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.328239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.327844    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.328484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.328237    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce5df295-143c-4805-aafc-5f700c804cab-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.328484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.328283    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.328484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.328378    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.328832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.328515    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce5df295-143c-4805-aafc-5f700c804cab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.329856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.329828    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.330010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.329974    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-config\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.330010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.330001    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.330675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.330653    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.330942 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.330923    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.330983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.330944    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce5df295-143c-4805-aafc-5f700c804cab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.332348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.332332    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgjq\" (UniqueName: \"kubernetes.io/projected/ce5df295-143c-4805-aafc-5f700c804cab-kube-api-access-txgjq\") pod \"prometheus-k8s-0\" (UID: \"ce5df295-143c-4805-aafc-5f700c804cab\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.524899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.524837    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:51.803952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.803867    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 10:29:51.807547 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:29:51.807517    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5df295_143c_4805_aafc_5f700c804cab.slice/crio-d4fe436516c7b69456a704869ba640a59c93bc41d0adec1cab319f6cd7635ba8 WatchSource:0}: Error finding container d4fe436516c7b69456a704869ba640a59c93bc41d0adec1cab319f6cd7635ba8: Status 404 returned error can't find the container with id d4fe436516c7b69456a704869ba640a59c93bc41d0adec1cab319f6cd7635ba8
Nov 03 10:29:51.898336 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:51.898309    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d34fc88-a058-4770-b5e7-dea808286e2d" path="/var/lib/kubelet/pods/9d34fc88-a058-4770-b5e7-dea808286e2d/volumes"
Nov 03 10:29:52.028808 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:52.028777    5817 generic.go:356] "Generic (PLEG): container finished" podID="ce5df295-143c-4805-aafc-5f700c804cab" containerID="85a1308b068d98bf8bd8b6c2b621a7ae9506004acb238ebd3d542e10a313aae6" exitCode=0
Nov 03 10:29:52.029211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:52.028817    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce5df295-143c-4805-aafc-5f700c804cab","Type":"ContainerDied","Data":"85a1308b068d98bf8bd8b6c2b621a7ae9506004acb238ebd3d542e10a313aae6"}
Nov 03 10:29:52.029211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:52.028882    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce5df295-143c-4805-aafc-5f700c804cab","Type":"ContainerStarted","Data":"d4fe436516c7b69456a704869ba640a59c93bc41d0adec1cab319f6cd7635ba8"}
Nov 03 10:29:52.410858 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:29:52.410818    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:29:53.035555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.035522    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce5df295-143c-4805-aafc-5f700c804cab","Type":"ContainerStarted","Data":"d43f712b543a376e0b61ef2cac73788216ecb95f158101fec743bb77a9f6556f"}
Nov 03 10:29:53.035965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.035607    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce5df295-143c-4805-aafc-5f700c804cab","Type":"ContainerStarted","Data":"c3d412184931d52681e504914a46182a106c755ea7e3c931abfb72ca5ae1a0d7"}
Nov 03 10:29:53.035965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.035633    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce5df295-143c-4805-aafc-5f700c804cab","Type":"ContainerStarted","Data":"3720c2ab8a841ba089beb1a317c0f6dcad3eedb18c2de1a39ead587b5f615370"}
Nov 03 10:29:53.035965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.035658    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce5df295-143c-4805-aafc-5f700c804cab","Type":"ContainerStarted","Data":"1e0ccf8fd139a9a1f0e508afdb3aeff1e75a47e992dd9249af1b2a993d90aafa"}
Nov 03 10:29:53.035965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.035680    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce5df295-143c-4805-aafc-5f700c804cab","Type":"ContainerStarted","Data":"ac07c03ae53afdb108833f4b14b1f53ee684dbdd9941a836e331083d241377f0"}
Nov 03 10:29:53.035965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.035701    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce5df295-143c-4805-aafc-5f700c804cab","Type":"ContainerStarted","Data":"28f31de9f20c8748ffe0e97e3f8cc8f73b6c2a66d0b07e982274fd60e001faad"}
Nov 03 10:29:53.063892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.063834    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.048761991 podStartE2EDuration="2.063820193s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.058202ms" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:53.06358954 +0000 UTC m=+23.269171625" watchObservedRunningTime="2026-11-03 10:29:53.063820193 +0000 UTC m=+23.269402253"
Nov 03 10:29:53.676032 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.675986    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:53.866238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:53.866205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:54.047882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:54.047838    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:54.182822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:54.182792    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:54.318355 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:54.318265    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:54.446107 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:54.446075    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:56.525850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:56.525822    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:29:56.998663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:56.998623    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:57.156193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:57.156164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:58.203108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:29:58.203081    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-brkhs"
Nov 03 10:30:22.410595 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:30:22.410539    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:30:28.365550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:28.365513    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:28.605293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:28.605264    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:30.721259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:30.721220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:38.381635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:38.381600    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:30:38.397117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:38.397097    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:30:39.204631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:39.204607    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 10:30:39.713062 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:39.713029    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:39.838814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:39.838765    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:44.518541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:44.518510    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:45.771482 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:45.771442    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:45.897463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:45.897429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:46.231686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:46.231645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:46.359291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:46.359256    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:52.301110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:52.301069    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:52.411790 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:30:52.411758    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:30:52.457143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:52.457112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:56.992117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:56.992083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:30:57.125606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:30:57.125561    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:31:19.511014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:31:19.510932    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:31:22.410547 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:31:22.410508    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:31:30.837832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:31:30.837791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:31:52.411708 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:31:52.411667    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:31:55.510317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:31:55.510284    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:32:22.409879 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:32:22.409844    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:32:30.807009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:32:30.806979    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:32:52.410363 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:32:52.410323    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:33:20.122017 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:20.121973    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:22.077011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:22.076979    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:22.410688 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:33:22.410525    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:33:30.856874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:30.856842    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:30.987797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:30.987745    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:32.100703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:32.100670    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:32.269481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:32.269447    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:33.388394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.388357    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:33.535480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.535449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:33.898396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.898368    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"]
Nov 03 10:33:33.901359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.901343    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:33.907790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.907161    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 10:33:33.908515 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.908492    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 10:33:33.908771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.908751    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-5n4pf\""
Nov 03 10:33:33.909237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.909219    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"]
Nov 03 10:33:33.949012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.948984    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrj2\" (UniqueName: \"kubernetes.io/projected/4eff77a4-3c4c-4b86-a76e-875e683631bd-kube-api-access-pcrj2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:33.949126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.949037    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:33.949126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.949098    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:33.996229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.996200    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"]
Nov 03 10:33:33.999409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:33.999391    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.002843 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.002823    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-p2mcx\""
Nov 03 10:33:34.007875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.007855    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"]
Nov 03 10:33:34.050291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.050266    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.050387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.050301    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:34.050387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.050329    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:34.050387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.050371    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.050492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.050393    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w588w\" (UniqueName: \"kubernetes.io/projected/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-kube-api-access-w588w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.050492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.050451    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrj2\" (UniqueName: \"kubernetes.io/projected/4eff77a4-3c4c-4b86-a76e-875e683631bd-kube-api-access-pcrj2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:34.050634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.050616    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:34.050674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.050650    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:34.058771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.058755    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrj2\" (UniqueName: \"kubernetes.io/projected/4eff77a4-3c4c-4b86-a76e-875e683631bd-kube-api-access-pcrj2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:34.094166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.094145    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"]
Nov 03 10:33:34.097287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.097273    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.100528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.100509    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5ztkx\""
Nov 03 10:33:34.105373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.105350    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"]
Nov 03 10:33:34.151358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.151307    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.151358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.151336    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w588w\" (UniqueName: \"kubernetes.io/projected/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-kube-api-access-w588w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.151480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.151370    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92p4l\" (UniqueName: \"kubernetes.io/projected/c1bc56f3-1075-4ac7-9675-c8be38dc1054-kube-api-access-92p4l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.151480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.151426    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.151480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.151454    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.151630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.151485    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.151710 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.151694    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.151767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.151751    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.159363 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.159343    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w588w\" (UniqueName: \"kubernetes.io/projected/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-kube-api-access-w588w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.196625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.196602    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"]
Nov 03 10:33:34.201279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.201253    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.207007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.206990    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-2lvz8\""
Nov 03 10:33:34.207668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.207647    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"]
Nov 03 10:33:34.217071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.217054    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:33:34.252741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.252701    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.252875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.252760    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92p4l\" (UniqueName: \"kubernetes.io/projected/c1bc56f3-1075-4ac7-9675-c8be38dc1054-kube-api-access-92p4l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.252875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.252855    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.253007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.252891    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/564b0ac9-0744-4cf0-97cc-f67c9c23e124-kube-api-access-lc7x7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.253007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.252912    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.253007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.252938    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.253303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.253280    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.253303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.253297    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.261398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.261376    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92p4l\" (UniqueName: \"kubernetes.io/projected/c1bc56f3-1075-4ac7-9675-c8be38dc1054-kube-api-access-92p4l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.315639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.315600    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:33:34.353546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.353476    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/564b0ac9-0744-4cf0-97cc-f67c9c23e124-kube-api-access-lc7x7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.353546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.353532    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.353773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.353600    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.354041 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.354018    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.354083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.354042    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.364737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.362316    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/564b0ac9-0744-4cf0-97cc-f67c9c23e124-kube-api-access-lc7x7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.390349 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.390322    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:34.413127 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.413072    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:33:34.490262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.490237    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"]
Nov 03 10:33:34.493243 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:34.493215    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eff77a4_3c4c_4b86_a76e_875e683631bd.slice/crio-29a6cebe290f95ffa0c0a7d9edfbeabc947a1b6cae1183873dcf1b8cbcaa8a25 WatchSource:0}: Error finding container 29a6cebe290f95ffa0c0a7d9edfbeabc947a1b6cae1183873dcf1b8cbcaa8a25: Status 404 returned error can't find the container with id 29a6cebe290f95ffa0c0a7d9edfbeabc947a1b6cae1183873dcf1b8cbcaa8a25
Nov 03 10:33:34.495192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.495174    5817 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:33:34.520773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.520734    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:33:34.588380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.588347    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"]
Nov 03 10:33:34.592944 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:34.592919    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df10d9d_8f18_41d9_9cc4_0a5bf45e256c.slice/crio-bdf0df5a143e218810e176cc738066e08fe0d4e4bef54c8d7b924a534deb66c4 WatchSource:0}: Error finding container bdf0df5a143e218810e176cc738066e08fe0d4e4bef54c8d7b924a534deb66c4: Status 404 returned error can't find the container with id bdf0df5a143e218810e176cc738066e08fe0d4e4bef54c8d7b924a534deb66c4
Nov 03 10:33:34.680220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.680193    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"]
Nov 03 10:33:34.682882 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:34.682854    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bc56f3_1075_4ac7_9675_c8be38dc1054.slice/crio-6eac7d83f67a64a52db0de009c2657c7b6ef52ba3fab5c44ddaf0b8fa7523949 WatchSource:0}: Error finding container 6eac7d83f67a64a52db0de009c2657c7b6ef52ba3fab5c44ddaf0b8fa7523949: Status 404 returned error can't find the container with id 6eac7d83f67a64a52db0de009c2657c7b6ef52ba3fab5c44ddaf0b8fa7523949
Nov 03 10:33:34.733839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.733803    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq" event={"ID":"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c","Type":"ContainerStarted","Data":"bdf0df5a143e218810e176cc738066e08fe0d4e4bef54c8d7b924a534deb66c4"}
Nov 03 10:33:34.734880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.734859    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk" event={"ID":"c1bc56f3-1075-4ac7-9675-c8be38dc1054","Type":"ContainerStarted","Data":"6eac7d83f67a64a52db0de009c2657c7b6ef52ba3fab5c44ddaf0b8fa7523949"}
Nov 03 10:33:34.735951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.735935    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t" event={"ID":"4eff77a4-3c4c-4b86-a76e-875e683631bd","Type":"ContainerStarted","Data":"29a6cebe290f95ffa0c0a7d9edfbeabc947a1b6cae1183873dcf1b8cbcaa8a25"}
Nov 03 10:33:34.786874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:34.786853    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"]
Nov 03 10:33:34.789426 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:34.789405    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564b0ac9_0744_4cf0_97cc_f67c9c23e124.slice/crio-8e7d8bc902305be0fe9407a173a9649183737bdde7d7a09fc52f0ec6a47c90aa WatchSource:0}: Error finding container 8e7d8bc902305be0fe9407a173a9649183737bdde7d7a09fc52f0ec6a47c90aa: Status 404 returned error can't find the container with id 8e7d8bc902305be0fe9407a173a9649183737bdde7d7a09fc52f0ec6a47c90aa
Nov 03 10:33:35.725502 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.725467    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"]
Nov 03 10:33:35.728950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.728926    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.733734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.733711    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qs7pn\""
Nov 03 10:33:35.737932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.737809    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"]
Nov 03 10:33:35.747879 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.747852    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw" event={"ID":"564b0ac9-0744-4cf0-97cc-f67c9c23e124","Type":"ContainerStarted","Data":"8e7d8bc902305be0fe9407a173a9649183737bdde7d7a09fc52f0ec6a47c90aa"}
Nov 03 10:33:35.764714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.764680    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7gm\" (UniqueName: \"kubernetes.io/projected/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-kube-api-access-8g7gm\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.764878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.764775    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.764878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.764810    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.865853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.865818    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.866073 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.865867    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.866073 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.865916    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7gm\" (UniqueName: \"kubernetes.io/projected/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-kube-api-access-8g7gm\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.866645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.866562    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.866854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.866833    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:35.879272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:35.879152    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7gm\" (UniqueName: \"kubernetes.io/projected/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-kube-api-access-8g7gm\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:36.055521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.055428    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:33:36.353821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.351785    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"]
Nov 03 10:33:36.360710 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:36.360658    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2c724a_3cb2_4b7d_961f_b9e51a1dcf17.slice/crio-59b41388e65dac37d0d75ddaea029042e5cb4096e853bd7c12780ae3cf564537 WatchSource:0}: Error finding container 59b41388e65dac37d0d75ddaea029042e5cb4096e853bd7c12780ae3cf564537: Status 404 returned error can't find the container with id 59b41388e65dac37d0d75ddaea029042e5cb4096e853bd7c12780ae3cf564537
Nov 03 10:33:36.439136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.437806    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"]
Nov 03 10:33:36.442335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.442307    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.451059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.451032    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-s4hh8\""
Nov 03 10:33:36.453844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.453787    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"]
Nov 03 10:33:36.572886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.572850    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8m9\" (UniqueName: \"kubernetes.io/projected/cb513550-94cd-4aa9-b207-87c213e801e5-kube-api-access-rn8m9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.573076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.572965    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.573076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.573016    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.675428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.674281    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.675428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.674336    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8m9\" (UniqueName: \"kubernetes.io/projected/cb513550-94cd-4aa9-b207-87c213e801e5-kube-api-access-rn8m9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.675428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.674427    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.675428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.674852    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.675428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.675074    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.687275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.687224    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8m9\" (UniqueName: \"kubernetes.io/projected/cb513550-94cd-4aa9-b207-87c213e801e5-kube-api-access-rn8m9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:36.754554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.754516    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw" event={"ID":"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17","Type":"ContainerStarted","Data":"59b41388e65dac37d0d75ddaea029042e5cb4096e853bd7c12780ae3cf564537"}
Nov 03 10:33:36.766934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:36.766876    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:33:37.060166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:37.060136    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"]
Nov 03 10:33:37.064280 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:37.064245    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb513550_94cd_4aa9_b207_87c213e801e5.slice/crio-32116ca02b2f9ecfca50db10cf3f7133d17ec399b19cf38c044b0bd46ed1131f WatchSource:0}: Error finding container 32116ca02b2f9ecfca50db10cf3f7133d17ec399b19cf38c044b0bd46ed1131f: Status 404 returned error can't find the container with id 32116ca02b2f9ecfca50db10cf3f7133d17ec399b19cf38c044b0bd46ed1131f
Nov 03 10:33:37.761500 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:37.761457    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx" event={"ID":"cb513550-94cd-4aa9-b207-87c213e801e5","Type":"ContainerStarted","Data":"32116ca02b2f9ecfca50db10cf3f7133d17ec399b19cf38c044b0bd46ed1131f"}
Nov 03 10:33:38.155595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.155348    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:38.297302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.297270    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:38.846780 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.845054    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"]
Nov 03 10:33:38.849598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.849521    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.859792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.858852    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 10:33:38.859792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.858954    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 10:33:38.859792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.858999    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rhbxr\""
Nov 03 10:33:38.859792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.859470    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"]
Nov 03 10:33:38.859792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.859748    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 10:33:38.894913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.894758    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.894913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.894801    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2z4w\" (UniqueName: \"kubernetes.io/projected/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-kube-api-access-h2z4w\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.894913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.894839    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.895166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.894972    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.895166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.895016    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.895166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.895054    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.895166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.895099    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.895166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.895130    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.895166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.895156    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.930342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.929359    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"]
Nov 03 10:33:38.933601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.933544    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.945718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.945630    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"]
Nov 03 10:33:38.957691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.957621    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-cjkzn\""
Nov 03 10:33:38.959852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.959824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996240    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996290    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996320    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996353    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996387    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996412    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996445    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996475    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996513    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996543    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996594    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996619    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996649    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996693    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996731    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92qq\" (UniqueName: \"kubernetes.io/projected/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-kube-api-access-m92qq\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996769    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996805    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.996833    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2z4w\" (UniqueName: \"kubernetes.io/projected/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-kube-api-access-h2z4w\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.997194    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.997374    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.997599    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.998055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.998007    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:38.999424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:38.998647    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:39.000940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.000914    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:39.001257 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.001235    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:39.011666 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.011621    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2z4w\" (UniqueName: \"kubernetes.io/projected/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-kube-api-access-h2z4w\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:39.012709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.012683    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf3ef4f-1840-4e68-9049-ec8a195d5e9b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-c7n4c\" (UID: \"edf3ef4f-1840-4e68-9049-ec8a195d5e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:39.098276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098208    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098246    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098275    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098298    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098340    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098367    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098396    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92qq\" (UniqueName: \"kubernetes.io/projected/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-kube-api-access-m92qq\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098422    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098444    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098687    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.098865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.098793    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.100043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.099219    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.100043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.099887    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.100043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.099941    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.102386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.102365    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.102938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.102912    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.115102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.115046    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92qq\" (UniqueName: \"kubernetes.io/projected/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-kube-api-access-m92qq\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.123434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.123353    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48\" (UID: \"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.131789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.131697    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:39.180845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.180412    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:39.240751 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.239321    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"]
Nov 03 10:33:39.248613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.247988    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.253309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.253287    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"]
Nov 03 10:33:39.253859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.253564    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:39.254461 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.253766    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8knf5\""
Nov 03 10:33:39.301999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.301455    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7jd\" (UniqueName: \"kubernetes.io/projected/cc65d628-9b6b-43ec-83cd-b0d0fec15195-kube-api-access-gb7jd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.302427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.302211    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.302427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.302286    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.402997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.402964    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.403167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.403119    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7jd\" (UniqueName: \"kubernetes.io/projected/cc65d628-9b6b-43ec-83cd-b0d0fec15195-kube-api-access-gb7jd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.403292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.403245    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.403810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.403774    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.403939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.403914    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.419393 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.419331    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7jd\" (UniqueName: \"kubernetes.io/projected/cc65d628-9b6b-43ec-83cd-b0d0fec15195-kube-api-access-gb7jd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.542996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.542963    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"]
Nov 03 10:33:39.553486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.552893    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"]
Nov 03 10:33:39.555756 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:39.555719    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf3ef4f_1840_4e68_9049_ec8a195d5e9b.slice/crio-f5aeeed4b0fba91170a50e5048bdf43946f60808dc79bb44e7e66f4224b54ed0 WatchSource:0}: Error finding container f5aeeed4b0fba91170a50e5048bdf43946f60808dc79bb44e7e66f4224b54ed0: Status 404 returned error can't find the container with id f5aeeed4b0fba91170a50e5048bdf43946f60808dc79bb44e7e66f4224b54ed0
Nov 03 10:33:39.557455 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:39.557422    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ef3ba8_5695_4f8f_9b6f_6b836b52f7e9.slice/crio-e33192ac833a258089172e1dac8e080b1b7e0c843d3078c4d3f93ad0ae500bbf WatchSource:0}: Error finding container e33192ac833a258089172e1dac8e080b1b7e0c843d3078c4d3f93ad0ae500bbf: Status 404 returned error can't find the container with id e33192ac833a258089172e1dac8e080b1b7e0c843d3078c4d3f93ad0ae500bbf
Nov 03 10:33:39.577693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.577489    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:33:39.775267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.775179    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" event={"ID":"edf3ef4f-1840-4e68-9049-ec8a195d5e9b","Type":"ContainerStarted","Data":"f5aeeed4b0fba91170a50e5048bdf43946f60808dc79bb44e7e66f4224b54ed0"}
Nov 03 10:33:39.778436 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.778357    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48" event={"ID":"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9","Type":"ContainerStarted","Data":"e33192ac833a258089172e1dac8e080b1b7e0c843d3078c4d3f93ad0ae500bbf"}
Nov 03 10:33:39.883383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:39.883296    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"]
Nov 03 10:33:39.889551 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:39.889519    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc65d628_9b6b_43ec_83cd_b0d0fec15195.slice/crio-3df5f4efeb8aee6eb35e274aecb71ae4c641cbe50b517222e2fa15fa4ecd75bc WatchSource:0}: Error finding container 3df5f4efeb8aee6eb35e274aecb71ae4c641cbe50b517222e2fa15fa4ecd75bc: Status 404 returned error can't find the container with id 3df5f4efeb8aee6eb35e274aecb71ae4c641cbe50b517222e2fa15fa4ecd75bc
Nov 03 10:33:40.800948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:40.800818    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs" event={"ID":"cc65d628-9b6b-43ec-83cd-b0d0fec15195","Type":"ContainerStarted","Data":"3df5f4efeb8aee6eb35e274aecb71ae4c641cbe50b517222e2fa15fa4ecd75bc"}
Nov 03 10:33:41.374395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.372709    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"]
Nov 03 10:33:41.378984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.378493    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.391462 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.390540    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"]
Nov 03 10:33:41.395209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.393500    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"]
Nov 03 10:33:41.399736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.399716    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.406671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.406427    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"]
Nov 03 10:33:41.425420 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.425383    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.425561 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.425445    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq46c\" (UniqueName: \"kubernetes.io/projected/5660ccd9-123d-4de8-9d98-da67d0f0bafe-kube-api-access-dq46c\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.425561 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.425508    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.426806 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.426708    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"]
Nov 03 10:33:41.430645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.430625    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.436873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.436558    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"]
Nov 03 10:33:41.526300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526269    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.526473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526332    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cll8p\" (UniqueName: \"kubernetes.io/projected/5237060b-3f82-4aea-a86b-a39893e57014-kube-api-access-cll8p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.526473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526371    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.526473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526402    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq46c\" (UniqueName: \"kubernetes.io/projected/5660ccd9-123d-4de8-9d98-da67d0f0bafe-kube-api-access-dq46c\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.526473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526441    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.526709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526491    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sst92\" (UniqueName: \"kubernetes.io/projected/08bb3d79-e89a-4295-b089-afb06b5fd6d5-kube-api-access-sst92\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.526709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526524    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.526709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526549    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.526709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.526610    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.527233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.527048    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.527233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.527196    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.538773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.538726    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq46c\" (UniqueName: \"kubernetes.io/projected/5660ccd9-123d-4de8-9d98-da67d0f0bafe-kube-api-access-dq46c\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.629087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.629011    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cll8p\" (UniqueName: \"kubernetes.io/projected/5237060b-3f82-4aea-a86b-a39893e57014-kube-api-access-cll8p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.629087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.629065    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.629284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.629101    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.629284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.629146    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sst92\" (UniqueName: \"kubernetes.io/projected/08bb3d79-e89a-4295-b089-afb06b5fd6d5-kube-api-access-sst92\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.629284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.629177    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.629284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.629209    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.630244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.629629    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.630244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.629976    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.630244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.630204    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.630473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.630316    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.641007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.640955    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sst92\" (UniqueName: \"kubernetes.io/projected/08bb3d79-e89a-4295-b089-afb06b5fd6d5-kube-api-access-sst92\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:41.661555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.661498    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cll8p\" (UniqueName: \"kubernetes.io/projected/5237060b-3f82-4aea-a86b-a39893e57014-kube-api-access-cll8p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.702541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.702496    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:33:41.705877 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.705853    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:33:41.726685 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.726658    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:33:41.754952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:41.754823    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:33:42.137528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:42.137494    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"]
Nov 03 10:33:42.150168 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:42.148997    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5660ccd9_123d_4de8_9d98_da67d0f0bafe.slice/crio-1bcba030bc8feeceffdd897c81f5de0d2170e3aaf40ad8635b7bfba054860f98 WatchSource:0}: Error finding container 1bcba030bc8feeceffdd897c81f5de0d2170e3aaf40ad8635b7bfba054860f98: Status 404 returned error can't find the container with id 1bcba030bc8feeceffdd897c81f5de0d2170e3aaf40ad8635b7bfba054860f98
Nov 03 10:33:42.157627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:42.157603    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"]
Nov 03 10:33:42.168712 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:33:42.168686    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bb3d79_e89a_4295_b089_afb06b5fd6d5.slice/crio-c32a510b41b0823e0be2230118c75d11b76bf4228c9099dc087146b370e5e24b WatchSource:0}: Error finding container c32a510b41b0823e0be2230118c75d11b76bf4228c9099dc087146b370e5e24b: Status 404 returned error can't find the container with id c32a510b41b0823e0be2230118c75d11b76bf4228c9099dc087146b370e5e24b
Nov 03 10:33:42.192861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:42.192833    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"]
Nov 03 10:33:42.825491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:42.825438    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql" event={"ID":"08bb3d79-e89a-4295-b089-afb06b5fd6d5","Type":"ContainerStarted","Data":"c32a510b41b0823e0be2230118c75d11b76bf4228c9099dc087146b370e5e24b"}
Nov 03 10:33:42.836982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:42.836927    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z" event={"ID":"5237060b-3f82-4aea-a86b-a39893e57014","Type":"ContainerStarted","Data":"aefcbe0c878efe84cc19dae5433f28ece7333de7b98ada4df22749355ce6023a"}
Nov 03 10:33:42.843529 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:42.843499    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf" event={"ID":"5660ccd9-123d-4de8-9d98-da67d0f0bafe","Type":"ContainerStarted","Data":"1bcba030bc8feeceffdd897c81f5de0d2170e3aaf40ad8635b7bfba054860f98"}
Nov 03 10:33:50.295933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:50.295221    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:33:50.295933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:50.295306    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:33:50.295933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:50.295345    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:33:50.796832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:50.796792    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:33:50.796965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:50.796885    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:33:50.796965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:50.796927    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:33:50.939698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:50.938788    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48" event={"ID":"37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9","Type":"ContainerStarted","Data":"3cb2ef37f7744b43d273a9d0189de34ad58b2f2f15746f3fb3408af323d2b45d"}
Nov 03 10:33:50.966372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:50.966315    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48" podStartSLOduration=2.233068178 podStartE2EDuration="12.966296701s" totalImagesPullingTime="10.733228523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:33:50.965762369 +0000 UTC m=+261.171344420" watchObservedRunningTime="2026-11-03 10:33:50.966296701 +0000 UTC m=+261.171878756"
Nov 03 10:33:51.254351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:51.254294    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:51.261479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:51.260987    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:51.953050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:51.953006    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" event={"ID":"edf3ef4f-1840-4e68-9049-ec8a195d5e9b","Type":"ContainerStarted","Data":"32fdb7a299da075ba3fc1005c70a500fabd01e71fb8243fc9d2a32d1734bd6e9"}
Nov 03 10:33:51.953618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:51.953175    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:51.954054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:51.953998    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48"
Nov 03 10:33:51.975634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:51.974286    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" podStartSLOduration=2.740131626 podStartE2EDuration="13.974268022s" totalImagesPullingTime="11.234136396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:33:51.971844203 +0000 UTC m=+262.177426254" watchObservedRunningTime="2026-11-03 10:33:51.974268022 +0000 UTC m=+262.179850077"
Nov 03 10:33:52.181074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:52.181045    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:52.186995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:52.186970    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:52.451660 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:33:52.451627    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:33:52.962525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:52.962164    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:52.963527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:52.963191    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:33:53.970401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:53.970173    5817 generic.go:356] "Generic (PLEG): container finished" podID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerID="797a0686a231d3b78d143d62df874d3972437d1679f4ca2455b9540234d56ba7" exitCode=0
Nov 03 10:33:53.970401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:53.970268    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq" event={"ID":"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c","Type":"ContainerDied","Data":"797a0686a231d3b78d143d62df874d3972437d1679f4ca2455b9540234d56ba7"}
Nov 03 10:33:53.974189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:53.973634    5817 generic.go:356] "Generic (PLEG): container finished" podID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerID="f1d2d786052365b97cbdbf5fe3e55430bddcee2c8f49330cd278e7672c58aac1" exitCode=0
Nov 03 10:33:53.974189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:53.973736    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk" event={"ID":"c1bc56f3-1075-4ac7-9675-c8be38dc1054","Type":"ContainerDied","Data":"f1d2d786052365b97cbdbf5fe3e55430bddcee2c8f49330cd278e7672c58aac1"}
Nov 03 10:33:53.978459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:53.977628    5817 generic.go:356] "Generic (PLEG): container finished" podID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerID="2e00136f3e22f8ca224d13393b405d09378a88afd32c8a13206c43436fdf1b2e" exitCode=0
Nov 03 10:33:53.978459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:53.977723    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs" event={"ID":"cc65d628-9b6b-43ec-83cd-b0d0fec15195","Type":"ContainerDied","Data":"2e00136f3e22f8ca224d13393b405d09378a88afd32c8a13206c43436fdf1b2e"}
Nov 03 10:33:53.982690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:53.982667    5817 generic.go:356] "Generic (PLEG): container finished" podID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerID="9df0c6bd0f79e09f8c186c715df99b7b7bcdbf3fd760e7318f7c88ed03036346" exitCode=0
Nov 03 10:33:53.983472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:53.983447    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t" event={"ID":"4eff77a4-3c4c-4b86-a76e-875e683631bd","Type":"ContainerDied","Data":"9df0c6bd0f79e09f8c186c715df99b7b7bcdbf3fd760e7318f7c88ed03036346"}
Nov 03 10:33:54.988102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:54.988063    5817 generic.go:356] "Generic (PLEG): container finished" podID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerID="234d53d3eda5d2467c4d6e2488c78639b2ad0d136911ebc4ff6a41d6de178f1f" exitCode=0
Nov 03 10:33:54.988620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:54.988138    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf" event={"ID":"5660ccd9-123d-4de8-9d98-da67d0f0bafe","Type":"ContainerDied","Data":"234d53d3eda5d2467c4d6e2488c78639b2ad0d136911ebc4ff6a41d6de178f1f"}
Nov 03 10:33:55.993851 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:55.993823    5817 generic.go:356] "Generic (PLEG): container finished" podID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerID="c1849baeb2e6dccc5590e11e874bc77e502f41500b07c2e5d870165d275936f7" exitCode=0
Nov 03 10:33:55.994317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:55.993917    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw" event={"ID":"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17","Type":"ContainerDied","Data":"c1849baeb2e6dccc5590e11e874bc77e502f41500b07c2e5d870165d275936f7"}
Nov 03 10:33:55.996193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:55.996168    5817 generic.go:356] "Generic (PLEG): container finished" podID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerID="0005966469e91e592147bd85962e73d90b66726ac4efcc9e4cbacdf5e9620c31" exitCode=0
Nov 03 10:33:55.996292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:55.996234    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq" event={"ID":"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c","Type":"ContainerDied","Data":"0005966469e91e592147bd85962e73d90b66726ac4efcc9e4cbacdf5e9620c31"}
Nov 03 10:33:55.998717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:55.998697    5817 generic.go:356] "Generic (PLEG): container finished" podID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerID="0900f4f4d8facde94c3ccc03938fb7a4e441e8951b7690af60707059eaf94ee5" exitCode=0
Nov 03 10:33:55.998804 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:55.998763    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk" event={"ID":"c1bc56f3-1075-4ac7-9675-c8be38dc1054","Type":"ContainerDied","Data":"0900f4f4d8facde94c3ccc03938fb7a4e441e8951b7690af60707059eaf94ee5"}
Nov 03 10:33:56.000391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.000278    5817 generic.go:356] "Generic (PLEG): container finished" podID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerID="c79532c53ae419b9977caf6a42a3d5558e5a1d6991ea8416b487195117ea9d7d" exitCode=0
Nov 03 10:33:56.000391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.000306    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw" event={"ID":"564b0ac9-0744-4cf0-97cc-f67c9c23e124","Type":"ContainerDied","Data":"c79532c53ae419b9977caf6a42a3d5558e5a1d6991ea8416b487195117ea9d7d"}
Nov 03 10:33:56.001906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.001879    5817 generic.go:356] "Generic (PLEG): container finished" podID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerID="83293bbc4eb75b2d9e7e9e6192c8ff9a6c8a3a2f499cd867ba0e129fab84c5c3" exitCode=0
Nov 03 10:33:56.001994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.001963    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql" event={"ID":"08bb3d79-e89a-4295-b089-afb06b5fd6d5","Type":"ContainerDied","Data":"83293bbc4eb75b2d9e7e9e6192c8ff9a6c8a3a2f499cd867ba0e129fab84c5c3"}
Nov 03 10:33:56.003476 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.003458    5817 generic.go:356] "Generic (PLEG): container finished" podID="cb513550-94cd-4aa9-b207-87c213e801e5" containerID="d395a734b59f204b41891786004f8959a95bac35316c60f6fce6c52f25477c84" exitCode=0
Nov 03 10:33:56.003554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.003527    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx" event={"ID":"cb513550-94cd-4aa9-b207-87c213e801e5","Type":"ContainerDied","Data":"d395a734b59f204b41891786004f8959a95bac35316c60f6fce6c52f25477c84"}
Nov 03 10:33:56.005008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.004991    5817 generic.go:356] "Generic (PLEG): container finished" podID="5237060b-3f82-4aea-a86b-a39893e57014" containerID="f1cac56947091a29628b0e829399c3728a5b141e95def1e144a4f5218acae75c" exitCode=0
Nov 03 10:33:56.005103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.005055    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z" event={"ID":"5237060b-3f82-4aea-a86b-a39893e57014","Type":"ContainerDied","Data":"f1cac56947091a29628b0e829399c3728a5b141e95def1e144a4f5218acae75c"}
Nov 03 10:33:56.006946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:56.006923    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t" event={"ID":"4eff77a4-3c4c-4b86-a76e-875e683631bd","Type":"ContainerStarted","Data":"77efb482677c04962eb28f97581e3b5be60e99263afc9fee6e029194685093ec"}
Nov 03 10:33:57.016752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.016715    5817 generic.go:356] "Generic (PLEG): container finished" podID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerID="fc31c133d51ec91ef88defe8ef6dadb5b1ce699d7df91e0d6e31bea42f575883" exitCode=0
Nov 03 10:33:57.017220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.016821    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql" event={"ID":"08bb3d79-e89a-4295-b089-afb06b5fd6d5","Type":"ContainerDied","Data":"fc31c133d51ec91ef88defe8ef6dadb5b1ce699d7df91e0d6e31bea42f575883"}
Nov 03 10:33:57.023105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.023078    5817 generic.go:356] "Generic (PLEG): container finished" podID="5237060b-3f82-4aea-a86b-a39893e57014" containerID="51dd71db08ed2357797465a8688db2632ea43d106203917f67446d21cdff36b6" exitCode=0
Nov 03 10:33:57.023216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.023172    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z" event={"ID":"5237060b-3f82-4aea-a86b-a39893e57014","Type":"ContainerDied","Data":"51dd71db08ed2357797465a8688db2632ea43d106203917f67446d21cdff36b6"}
Nov 03 10:33:57.026654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.026621    5817 generic.go:356] "Generic (PLEG): container finished" podID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerID="77efb482677c04962eb28f97581e3b5be60e99263afc9fee6e029194685093ec" exitCode=0
Nov 03 10:33:57.026738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.026728    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t" event={"ID":"4eff77a4-3c4c-4b86-a76e-875e683631bd","Type":"ContainerDied","Data":"77efb482677c04962eb28f97581e3b5be60e99263afc9fee6e029194685093ec"}
Nov 03 10:33:57.029530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.029507    5817 generic.go:356] "Generic (PLEG): container finished" podID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerID="81671bfed1d1abe7328e0757ebaed8b9feb4dd3136b4f5d6c2d5ac8fdddb95ae" exitCode=0
Nov 03 10:33:57.029671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.029605    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw" event={"ID":"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17","Type":"ContainerDied","Data":"81671bfed1d1abe7328e0757ebaed8b9feb4dd3136b4f5d6c2d5ac8fdddb95ae"}
Nov 03 10:33:57.034907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.033414    5817 generic.go:356] "Generic (PLEG): container finished" podID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerID="882d57baddf92c791128455b541079e04dbf67c181f81868f5b391211f99d5c9" exitCode=0
Nov 03 10:33:57.034907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.033534    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf" event={"ID":"5660ccd9-123d-4de8-9d98-da67d0f0bafe","Type":"ContainerDied","Data":"882d57baddf92c791128455b541079e04dbf67c181f81868f5b391211f99d5c9"}
Nov 03 10:33:57.039114 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.038598    5817 generic.go:356] "Generic (PLEG): container finished" podID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerID="8279e335b7aeaa888072db55a5f7e5090e7801e2f2d870921d4aee6a27890995" exitCode=0
Nov 03 10:33:57.039114 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:33:57.038642    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw" event={"ID":"564b0ac9-0744-4cf0-97cc-f67c9c23e124","Type":"ContainerDied","Data":"8279e335b7aeaa888072db55a5f7e5090e7801e2f2d870921d4aee6a27890995"}
Nov 03 10:34:04.137602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:04.137546    5817 generic.go:356] "Generic (PLEG): container finished" podID="cb513550-94cd-4aa9-b207-87c213e801e5" containerID="1a49acdbe8ea835066b76d662467ce54af4646f40cca3b824a8aa6375d37ee1d" exitCode=0
Nov 03 10:34:04.138134 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:04.137676    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx" event={"ID":"cb513550-94cd-4aa9-b207-87c213e801e5","Type":"ContainerDied","Data":"1a49acdbe8ea835066b76d662467ce54af4646f40cca3b824a8aa6375d37ee1d"}
Nov 03 10:34:09.203045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:09.203000    5817 generic.go:356] "Generic (PLEG): container finished" podID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerID="cfbe0d2fb632c205b3de22dacfa73c2be81edc0f5763186ef6d56324d99f2ef8" exitCode=0
Nov 03 10:34:09.203563 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:09.203059    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs" event={"ID":"cc65d628-9b6b-43ec-83cd-b0d0fec15195","Type":"ContainerDied","Data":"cfbe0d2fb632c205b3de22dacfa73c2be81edc0f5763186ef6d56324d99f2ef8"}
Nov 03 10:34:15.270928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:15.270887    5817 generic.go:356] "Generic (PLEG): container finished" podID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerID="fda7449f41f54ed11d809ca2f7e8d7cf23617f3c8940bf05d91f97518fb5cf8a" exitCode=0
Nov 03 10:34:15.271463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:15.271028    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq" event={"ID":"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c","Type":"ContainerDied","Data":"fda7449f41f54ed11d809ca2f7e8d7cf23617f3c8940bf05d91f97518fb5cf8a"}
Nov 03 10:34:16.284735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.284705    5817 generic.go:356] "Generic (PLEG): container finished" podID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerID="9527c16cfc9ed0fb07ae28404d4ce451be8beb01b0ab8f93d697ff28304a82ad" exitCode=0
Nov 03 10:34:16.285110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.284846    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t" event={"ID":"4eff77a4-3c4c-4b86-a76e-875e683631bd","Type":"ContainerDied","Data":"9527c16cfc9ed0fb07ae28404d4ce451be8beb01b0ab8f93d697ff28304a82ad"}
Nov 03 10:34:16.291414 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.290955    5817 generic.go:356] "Generic (PLEG): container finished" podID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerID="efe56a1e75ad7bd882509823dd989692d6e5c1866fb7d42334d0aee284255632" exitCode=0
Nov 03 10:34:16.291414 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.291367    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk" event={"ID":"c1bc56f3-1075-4ac7-9675-c8be38dc1054","Type":"ContainerDied","Data":"efe56a1e75ad7bd882509823dd989692d6e5c1866fb7d42334d0aee284255632"}
Nov 03 10:34:16.481658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.481629    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:34:16.605427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.605391    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-kube-api-access-w588w\" (UniqueName: \"kubernetes.io/projected/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-kube-api-access-w588w\") pod \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") "
Nov 03 10:34:16.605656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.605495    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-bundle\") pod \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") "
Nov 03 10:34:16.605656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.605539    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-util\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-util\") pod \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\" (UID: \"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c\") "
Nov 03 10:34:16.606554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.606521    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-bundle" pod "4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" (UID: "4df10d9d-8f18-41d9-9cc4-0a5bf45e256c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:16.609932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.609892    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-kube-api-access-w588w" pod "4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" (UID: "4df10d9d-8f18-41d9-9cc4-0a5bf45e256c"). InnerVolumeSpecName "kube-api-access-w588w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:16.613286 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.613260    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-util" pod "4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" (UID: "4df10d9d-8f18-41d9-9cc4-0a5bf45e256c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:16.707143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.707107    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w588w\" (UniqueName: \"kubernetes.io/projected/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-kube-api-access-w588w\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:16.707143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.707145    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:16.707395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:16.707161    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4df10d9d-8f18-41d9-9cc4-0a5bf45e256c-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:17.303111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.300894    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq"
Nov 03 10:34:17.303111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.300908    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fhkhq" event={"ID":"4df10d9d-8f18-41d9-9cc4-0a5bf45e256c","Type":"ContainerDied","Data":"bdf0df5a143e218810e176cc738066e08fe0d4e4bef54c8d7b924a534deb66c4"}
Nov 03 10:34:17.303111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.300954    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf0df5a143e218810e176cc738066e08fe0d4e4bef54c8d7b924a534deb66c4"
Nov 03 10:34:17.304128 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.304098    5817 generic.go:356] "Generic (PLEG): container finished" podID="cb513550-94cd-4aa9-b207-87c213e801e5" containerID="120b60ea2db9b76ba827fe17180835eaf335da836e37bf1341a281c8e3c82078" exitCode=0
Nov 03 10:34:17.304245 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.304189    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx" event={"ID":"cb513550-94cd-4aa9-b207-87c213e801e5","Type":"ContainerDied","Data":"120b60ea2db9b76ba827fe17180835eaf335da836e37bf1341a281c8e3c82078"}
Nov 03 10:34:17.309599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.307139    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z" event={"ID":"5237060b-3f82-4aea-a86b-a39893e57014","Type":"ContainerStarted","Data":"070cb0b51b0b7a7f5fb3eea3a5be3b002295e2f79349875d0cbbd0e2fb5d1b2e"}
Nov 03 10:34:17.488697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.488669    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:34:17.514532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.514499    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:34:17.616205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.616169    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bc56f3-1075-4ac7-9675-c8be38dc1054-kube-api-access-92p4l\" (UniqueName: \"kubernetes.io/projected/c1bc56f3-1075-4ac7-9675-c8be38dc1054-kube-api-access-92p4l\") pod \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") "
Nov 03 10:34:17.616368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.616233    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-util\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-util\") pod \"4eff77a4-3c4c-4b86-a76e-875e683631bd\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") "
Nov 03 10:34:17.616368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.616300    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-util\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-util\") pod \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") "
Nov 03 10:34:17.616368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.616344    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-bundle\") pod \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\" (UID: \"c1bc56f3-1075-4ac7-9675-c8be38dc1054\") "
Nov 03 10:34:17.616545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.616378    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eff77a4-3c4c-4b86-a76e-875e683631bd-kube-api-access-pcrj2\" (UniqueName: \"kubernetes.io/projected/4eff77a4-3c4c-4b86-a76e-875e683631bd-kube-api-access-pcrj2\") pod \"4eff77a4-3c4c-4b86-a76e-875e683631bd\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") "
Nov 03 10:34:17.616545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.616420    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-bundle\") pod \"4eff77a4-3c4c-4b86-a76e-875e683631bd\" (UID: \"4eff77a4-3c4c-4b86-a76e-875e683631bd\") "
Nov 03 10:34:17.617258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.617229    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-bundle" pod "4eff77a4-3c4c-4b86-a76e-875e683631bd" (UID: "4eff77a4-3c4c-4b86-a76e-875e683631bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:17.618112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.618081    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-bundle" pod "c1bc56f3-1075-4ac7-9675-c8be38dc1054" (UID: "c1bc56f3-1075-4ac7-9675-c8be38dc1054"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:17.619157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.619128    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bc56f3-1075-4ac7-9675-c8be38dc1054-kube-api-access-92p4l" pod "c1bc56f3-1075-4ac7-9675-c8be38dc1054" (UID: "c1bc56f3-1075-4ac7-9675-c8be38dc1054"). InnerVolumeSpecName "kube-api-access-92p4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:17.624434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.624396    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-util" pod "c1bc56f3-1075-4ac7-9675-c8be38dc1054" (UID: "c1bc56f3-1075-4ac7-9675-c8be38dc1054"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:17.624783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.624750    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eff77a4-3c4c-4b86-a76e-875e683631bd-kube-api-access-pcrj2" pod "4eff77a4-3c4c-4b86-a76e-875e683631bd" (UID: "4eff77a4-3c4c-4b86-a76e-875e683631bd"). InnerVolumeSpecName "kube-api-access-pcrj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:17.624939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.624876    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-util" pod "4eff77a4-3c4c-4b86-a76e-875e683631bd" (UID: "4eff77a4-3c4c-4b86-a76e-875e683631bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:17.717933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.717904    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:17.718054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.717936    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1bc56f3-1075-4ac7-9675-c8be38dc1054-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:17.718054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.717952    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcrj2\" (UniqueName: \"kubernetes.io/projected/4eff77a4-3c4c-4b86-a76e-875e683631bd-kube-api-access-pcrj2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:17.718054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.717966    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:17.718054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.717983    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92p4l\" (UniqueName: \"kubernetes.io/projected/c1bc56f3-1075-4ac7-9675-c8be38dc1054-kube-api-access-92p4l\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:17.718054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:17.717995    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4eff77a4-3c4c-4b86-a76e-875e683631bd-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:18.315532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.315494    5817 generic.go:356] "Generic (PLEG): container finished" podID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerID="a187d10fa2db372751ffcd876b0b75f90430ead217156ff11a292e17defda7c8" exitCode=0
Nov 03 10:34:18.316036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.315600    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw" event={"ID":"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17","Type":"ContainerDied","Data":"a187d10fa2db372751ffcd876b0b75f90430ead217156ff11a292e17defda7c8"}
Nov 03 10:34:18.317986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.317962    5817 generic.go:356] "Generic (PLEG): container finished" podID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerID="f0534285e69e2d7c8a7603e4c2d7d44b81a5f5eff47535ee1a9251493f6ce222" exitCode=0
Nov 03 10:34:18.318096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.318054    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf" event={"ID":"5660ccd9-123d-4de8-9d98-da67d0f0bafe","Type":"ContainerDied","Data":"f0534285e69e2d7c8a7603e4c2d7d44b81a5f5eff47535ee1a9251493f6ce222"}
Nov 03 10:34:18.320465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.320446    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk" event={"ID":"c1bc56f3-1075-4ac7-9675-c8be38dc1054","Type":"ContainerDied","Data":"6eac7d83f67a64a52db0de009c2657c7b6ef52ba3fab5c44ddaf0b8fa7523949"}
Nov 03 10:34:18.320591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.320465    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5n7fk"
Nov 03 10:34:18.320591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.320485    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eac7d83f67a64a52db0de009c2657c7b6ef52ba3fab5c44ddaf0b8fa7523949"
Nov 03 10:34:18.323213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.323188    5817 generic.go:356] "Generic (PLEG): container finished" podID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerID="21380b5b5eccc99aaa165d8c9f7dace923d07793319d888a0fee45a7c3948843" exitCode=0
Nov 03 10:34:18.323316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.323271    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw" event={"ID":"564b0ac9-0744-4cf0-97cc-f67c9c23e124","Type":"ContainerDied","Data":"21380b5b5eccc99aaa165d8c9f7dace923d07793319d888a0fee45a7c3948843"}
Nov 03 10:34:18.325797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.325774    5817 generic.go:356] "Generic (PLEG): container finished" podID="5237060b-3f82-4aea-a86b-a39893e57014" containerID="070cb0b51b0b7a7f5fb3eea3a5be3b002295e2f79349875d0cbbd0e2fb5d1b2e" exitCode=0
Nov 03 10:34:18.325907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.325882    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z" event={"ID":"5237060b-3f82-4aea-a86b-a39893e57014","Type":"ContainerDied","Data":"070cb0b51b0b7a7f5fb3eea3a5be3b002295e2f79349875d0cbbd0e2fb5d1b2e"}
Nov 03 10:34:18.328050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.328033    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t"
Nov 03 10:34:18.328374 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.328362    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7rl4t" event={"ID":"4eff77a4-3c4c-4b86-a76e-875e683631bd","Type":"ContainerDied","Data":"29a6cebe290f95ffa0c0a7d9edfbeabc947a1b6cae1183873dcf1b8cbcaa8a25"}
Nov 03 10:34:18.328416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.328389    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a6cebe290f95ffa0c0a7d9edfbeabc947a1b6cae1183873dcf1b8cbcaa8a25"
Nov 03 10:34:18.467085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.467065    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:34:18.524310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.524290    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-util\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-util\") pod \"5237060b-3f82-4aea-a86b-a39893e57014\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") "
Nov 03 10:34:18.524364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.524350    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5237060b-3f82-4aea-a86b-a39893e57014-kube-api-access-cll8p\" (UniqueName: \"kubernetes.io/projected/5237060b-3f82-4aea-a86b-a39893e57014-kube-api-access-cll8p\") pod \"5237060b-3f82-4aea-a86b-a39893e57014\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") "
Nov 03 10:34:18.524414 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.524403    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-bundle\") pod \"5237060b-3f82-4aea-a86b-a39893e57014\" (UID: \"5237060b-3f82-4aea-a86b-a39893e57014\") "
Nov 03 10:34:18.525152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.525125    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-bundle" pod "5237060b-3f82-4aea-a86b-a39893e57014" (UID: "5237060b-3f82-4aea-a86b-a39893e57014"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:18.527530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.527486    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5237060b-3f82-4aea-a86b-a39893e57014-kube-api-access-cll8p" pod "5237060b-3f82-4aea-a86b-a39893e57014" (UID: "5237060b-3f82-4aea-a86b-a39893e57014"). InnerVolumeSpecName "kube-api-access-cll8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:18.530665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.530455    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-util" pod "5237060b-3f82-4aea-a86b-a39893e57014" (UID: "5237060b-3f82-4aea-a86b-a39893e57014"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:18.532601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.532496    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:34:18.625088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.625064    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb513550-94cd-4aa9-b207-87c213e801e5-kube-api-access-rn8m9\" (UniqueName: \"kubernetes.io/projected/cb513550-94cd-4aa9-b207-87c213e801e5-kube-api-access-rn8m9\") pod \"cb513550-94cd-4aa9-b207-87c213e801e5\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") "
Nov 03 10:34:18.625178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.625134    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-bundle\") pod \"cb513550-94cd-4aa9-b207-87c213e801e5\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") "
Nov 03 10:34:18.625228 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.625216    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-util\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-util\") pod \"cb513550-94cd-4aa9-b207-87c213e801e5\" (UID: \"cb513550-94cd-4aa9-b207-87c213e801e5\") "
Nov 03 10:34:18.625408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.625391    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:18.625408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.625408    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cll8p\" (UniqueName: \"kubernetes.io/projected/5237060b-3f82-4aea-a86b-a39893e57014-kube-api-access-cll8p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:18.625595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.625527    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-bundle" pod "cb513550-94cd-4aa9-b207-87c213e801e5" (UID: "cb513550-94cd-4aa9-b207-87c213e801e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:18.625595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.625556    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5237060b-3f82-4aea-a86b-a39893e57014-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:18.627268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.627247    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb513550-94cd-4aa9-b207-87c213e801e5-kube-api-access-rn8m9" pod "cb513550-94cd-4aa9-b207-87c213e801e5" (UID: "cb513550-94cd-4aa9-b207-87c213e801e5"). InnerVolumeSpecName "kube-api-access-rn8m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:18.630382 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.630363    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-util" pod "cb513550-94cd-4aa9-b207-87c213e801e5" (UID: "cb513550-94cd-4aa9-b207-87c213e801e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:18.726385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.726356    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:18.726385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.726389    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8m9\" (UniqueName: \"kubernetes.io/projected/cb513550-94cd-4aa9-b207-87c213e801e5-kube-api-access-rn8m9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:18.726597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:18.726404    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb513550-94cd-4aa9-b207-87c213e801e5-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.334297 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.334266    5817 generic.go:356] "Generic (PLEG): container finished" podID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerID="d0273a5da325a005e72f995612a4646d1a8e5f21df9f36893c8dec7db146b359" exitCode=0
Nov 03 10:34:19.334713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.334352    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs" event={"ID":"cc65d628-9b6b-43ec-83cd-b0d0fec15195","Type":"ContainerDied","Data":"d0273a5da325a005e72f995612a4646d1a8e5f21df9f36893c8dec7db146b359"}
Nov 03 10:34:19.336521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.336502    5817 generic.go:356] "Generic (PLEG): container finished" podID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerID="b098718b41a264b03717002941d61e1d1d5d57ad64153424463b0c6f9fedfb35" exitCode=0
Nov 03 10:34:19.336635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.336598    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql" event={"ID":"08bb3d79-e89a-4295-b089-afb06b5fd6d5","Type":"ContainerDied","Data":"b098718b41a264b03717002941d61e1d1d5d57ad64153424463b0c6f9fedfb35"}
Nov 03 10:34:19.338627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.338603    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx" event={"ID":"cb513550-94cd-4aa9-b207-87c213e801e5","Type":"ContainerDied","Data":"32116ca02b2f9ecfca50db10cf3f7133d17ec399b19cf38c044b0bd46ed1131f"}
Nov 03 10:34:19.338735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.338646    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32116ca02b2f9ecfca50db10cf3f7133d17ec399b19cf38c044b0bd46ed1131f"
Nov 03 10:34:19.338735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.338612    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64498kkx"
Nov 03 10:34:19.340514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.340496    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z"
Nov 03 10:34:19.340650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.340522    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c866z" event={"ID":"5237060b-3f82-4aea-a86b-a39893e57014","Type":"ContainerDied","Data":"aefcbe0c878efe84cc19dae5433f28ece7333de7b98ada4df22749355ce6023a"}
Nov 03 10:34:19.340650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.340594    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aefcbe0c878efe84cc19dae5433f28ece7333de7b98ada4df22749355ce6023a"
Nov 03 10:34:19.475173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.475151    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:34:19.504653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.504623    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:34:19.524421 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.524395    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:34:19.533959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.533934    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-bundle\") pod \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") "
Nov 03 10:34:19.534103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.533986    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-util\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-util\") pod \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") "
Nov 03 10:34:19.534103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.534049    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-kube-api-access-8g7gm\" (UniqueName: \"kubernetes.io/projected/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-kube-api-access-8g7gm\") pod \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\" (UID: \"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17\") "
Nov 03 10:34:19.534530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.534501    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-bundle" pod "bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" (UID: "bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:19.536239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.536213    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-kube-api-access-8g7gm" pod "bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" (UID: "bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17"). InnerVolumeSpecName "kube-api-access-8g7gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:19.539802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.539780    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-util" pod "bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" (UID: "bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:19.635295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635212    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-util\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-util\") pod \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") "
Nov 03 10:34:19.635295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635256    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-bundle\") pod \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") "
Nov 03 10:34:19.635295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635289    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5660ccd9-123d-4de8-9d98-da67d0f0bafe-kube-api-access-dq46c\" (UniqueName: \"kubernetes.io/projected/5660ccd9-123d-4de8-9d98-da67d0f0bafe-kube-api-access-dq46c\") pod \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") "
Nov 03 10:34:19.635598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635356    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-util\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-util\") pod \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\" (UID: \"5660ccd9-123d-4de8-9d98-da67d0f0bafe\") "
Nov 03 10:34:19.635598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635383    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-bundle\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-bundle\") pod \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") "
Nov 03 10:34:19.635598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635433    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564b0ac9-0744-4cf0-97cc-f67c9c23e124-kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/564b0ac9-0744-4cf0-97cc-f67c9c23e124-kube-api-access-lc7x7\") pod \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\" (UID: \"564b0ac9-0744-4cf0-97cc-f67c9c23e124\") "
Nov 03 10:34:19.635759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635662    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.635759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635682    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7gm\" (UniqueName: \"kubernetes.io/projected/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-kube-api-access-8g7gm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.635759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635697    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.635901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635817    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-bundle" pod "5660ccd9-123d-4de8-9d98-da67d0f0bafe" (UID: "5660ccd9-123d-4de8-9d98-da67d0f0bafe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:19.635958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.635911    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-bundle" pod "564b0ac9-0744-4cf0-97cc-f67c9c23e124" (UID: "564b0ac9-0744-4cf0-97cc-f67c9c23e124"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:19.637531 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.637505    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564b0ac9-0744-4cf0-97cc-f67c9c23e124-kube-api-access-lc7x7" pod "564b0ac9-0744-4cf0-97cc-f67c9c23e124" (UID: "564b0ac9-0744-4cf0-97cc-f67c9c23e124"). InnerVolumeSpecName "kube-api-access-lc7x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:19.637652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.637563    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5660ccd9-123d-4de8-9d98-da67d0f0bafe-kube-api-access-dq46c" pod "5660ccd9-123d-4de8-9d98-da67d0f0bafe" (UID: "5660ccd9-123d-4de8-9d98-da67d0f0bafe"). InnerVolumeSpecName "kube-api-access-dq46c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:19.641533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.641508    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-util" pod "564b0ac9-0744-4cf0-97cc-f67c9c23e124" (UID: "564b0ac9-0744-4cf0-97cc-f67c9c23e124"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:19.641533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.641516    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-util" pod "5660ccd9-123d-4de8-9d98-da67d0f0bafe" (UID: "5660ccd9-123d-4de8-9d98-da67d0f0bafe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:19.736831 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.736804    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.736831 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.736828    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.736965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.736838    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq46c\" (UniqueName: \"kubernetes.io/projected/5660ccd9-123d-4de8-9d98-da67d0f0bafe-kube-api-access-dq46c\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.736965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.736847    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5660ccd9-123d-4de8-9d98-da67d0f0bafe-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.736965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.736856    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/564b0ac9-0744-4cf0-97cc-f67c9c23e124-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:19.736965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:19.736864    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/564b0ac9-0744-4cf0-97cc-f67c9c23e124-kube-api-access-lc7x7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:20.346676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.346646    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw"
Nov 03 10:34:20.346676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.346665    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qb9pw" event={"ID":"564b0ac9-0744-4cf0-97cc-f67c9c23e124","Type":"ContainerDied","Data":"8e7d8bc902305be0fe9407a173a9649183737bdde7d7a09fc52f0ec6a47c90aa"}
Nov 03 10:34:20.347138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.346713    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7d8bc902305be0fe9407a173a9649183737bdde7d7a09fc52f0ec6a47c90aa"
Nov 03 10:34:20.348819 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.348791    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw"
Nov 03 10:34:20.348948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.348785    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xxpcw" event={"ID":"bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17","Type":"ContainerDied","Data":"59b41388e65dac37d0d75ddaea029042e5cb4096e853bd7c12780ae3cf564537"}
Nov 03 10:34:20.348948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.348910    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59b41388e65dac37d0d75ddaea029042e5cb4096e853bd7c12780ae3cf564537"
Nov 03 10:34:20.350932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.350903    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf" event={"ID":"5660ccd9-123d-4de8-9d98-da67d0f0bafe","Type":"ContainerDied","Data":"1bcba030bc8feeceffdd897c81f5de0d2170e3aaf40ad8635b7bfba054860f98"}
Nov 03 10:34:20.351018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.350949    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bcba030bc8feeceffdd897c81f5de0d2170e3aaf40ad8635b7bfba054860f98"
Nov 03 10:34:20.351018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.350965    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6556nf"
Nov 03 10:34:20.483904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.483878    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:34:20.492380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.492363    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:34:20.543609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.543550    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-util\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-util\") pod \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") "
Nov 03 10:34:20.543609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.543602    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-bundle\") pod \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") "
Nov 03 10:34:20.543816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.543648    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc65d628-9b6b-43ec-83cd-b0d0fec15195-kube-api-access-gb7jd\" (UniqueName: \"kubernetes.io/projected/cc65d628-9b6b-43ec-83cd-b0d0fec15195-kube-api-access-gb7jd\") pod \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") "
Nov 03 10:34:20.543816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.543680    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bb3d79-e89a-4295-b089-afb06b5fd6d5-kube-api-access-sst92\" (UniqueName: \"kubernetes.io/projected/08bb3d79-e89a-4295-b089-afb06b5fd6d5-kube-api-access-sst92\") pod \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") "
Nov 03 10:34:20.543816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.543734    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-bundle\") pod \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\" (UID: \"cc65d628-9b6b-43ec-83cd-b0d0fec15195\") "
Nov 03 10:34:20.543816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.543772    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-util\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-util\") pod \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\" (UID: \"08bb3d79-e89a-4295-b089-afb06b5fd6d5\") "
Nov 03 10:34:20.544618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.544590    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-bundle" pod "cc65d628-9b6b-43ec-83cd-b0d0fec15195" (UID: "cc65d628-9b6b-43ec-83cd-b0d0fec15195"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:20.544732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.544623    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-bundle" pod "08bb3d79-e89a-4295-b089-afb06b5fd6d5" (UID: "08bb3d79-e89a-4295-b089-afb06b5fd6d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:20.545920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.545900    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc65d628-9b6b-43ec-83cd-b0d0fec15195-kube-api-access-gb7jd" pod "cc65d628-9b6b-43ec-83cd-b0d0fec15195" (UID: "cc65d628-9b6b-43ec-83cd-b0d0fec15195"). InnerVolumeSpecName "kube-api-access-gb7jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:20.545993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.545943    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bb3d79-e89a-4295-b089-afb06b5fd6d5-kube-api-access-sst92" pod "08bb3d79-e89a-4295-b089-afb06b5fd6d5" (UID: "08bb3d79-e89a-4295-b089-afb06b5fd6d5"). InnerVolumeSpecName "kube-api-access-sst92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:20.549921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.549898    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-util" pod "cc65d628-9b6b-43ec-83cd-b0d0fec15195" (UID: "cc65d628-9b6b-43ec-83cd-b0d0fec15195"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:20.549921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.549907    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-util" pod "08bb3d79-e89a-4295-b089-afb06b5fd6d5" (UID: "08bb3d79-e89a-4295-b089-afb06b5fd6d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:20.644967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.644910    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:20.644967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.644928    5817 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:20.644967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.644938    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08bb3d79-e89a-4295-b089-afb06b5fd6d5-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:20.644967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.644946    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7jd\" (UniqueName: \"kubernetes.io/projected/cc65d628-9b6b-43ec-83cd-b0d0fec15195-kube-api-access-gb7jd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:20.644967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.644956    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sst92\" (UniqueName: \"kubernetes.io/projected/08bb3d79-e89a-4295-b089-afb06b5fd6d5-kube-api-access-sst92\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:20.644967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:20.644964    5817 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc65d628-9b6b-43ec-83cd-b0d0fec15195-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:34:21.357593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:21.357544    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs" event={"ID":"cc65d628-9b6b-43ec-83cd-b0d0fec15195","Type":"ContainerDied","Data":"3df5f4efeb8aee6eb35e274aecb71ae4c641cbe50b517222e2fa15fa4ecd75bc"}
Nov 03 10:34:21.358038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:21.357614    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3df5f4efeb8aee6eb35e274aecb71ae4c641cbe50b517222e2fa15fa4ecd75bc"
Nov 03 10:34:21.358038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:21.357552    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nmzs"
Nov 03 10:34:21.359446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:21.359424    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql"
Nov 03 10:34:21.359446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:21.359437    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzqql" event={"ID":"08bb3d79-e89a-4295-b089-afb06b5fd6d5","Type":"ContainerDied","Data":"c32a510b41b0823e0be2230118c75d11b76bf4228c9099dc087146b370e5e24b"}
Nov 03 10:34:21.359599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:21.359468    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c32a510b41b0823e0be2230118c75d11b76bf4228c9099dc087146b370e5e24b"
Nov 03 10:34:22.409800 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:34:22.409754    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:34:22.801464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:22.801429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:22.940113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:22.940080    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:24.424755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.424724    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"]
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425110    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerName="util"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425126    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerName="util"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425139    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerName="pull"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425146    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerName="pull"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425155    5817 cpu_manager.go:414] "containerMap: removing container" podUID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerName="pull"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425162    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerName="pull"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425173    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerName="extract"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425179    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerName="extract"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425196    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerName="pull"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425204    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerName="pull"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425214    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerName="util"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425221    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerName="util"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425229    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerName="extract"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425235    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerName="extract"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425243    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerName="extract"
Nov 03 10:34:24.425243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425249    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425295    5817 cpu_manager.go:414] "containerMap: removing container" podUID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425303    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425319    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425326    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425337    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5237060b-3f82-4aea-a86b-a39893e57014" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425344    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5237060b-3f82-4aea-a86b-a39893e57014" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425356    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425362    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425371    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425377    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425387    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425394    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425402    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cb513550-94cd-4aa9-b207-87c213e801e5" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425409    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb513550-94cd-4aa9-b207-87c213e801e5" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425418    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425424    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425434    5817 cpu_manager.go:414] "containerMap: removing container" podUID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425440    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425450    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cb513550-94cd-4aa9-b207-87c213e801e5" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425456    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb513550-94cd-4aa9-b207-87c213e801e5" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425465    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425473    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425481    5817 cpu_manager.go:414] "containerMap: removing container" podUID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425488    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425502    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5237060b-3f82-4aea-a86b-a39893e57014" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425508    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5237060b-3f82-4aea-a86b-a39893e57014" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425518    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425525    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425534    5817 cpu_manager.go:414] "containerMap: removing container" podUID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425541    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425555    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425561    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425590    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425596    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425604    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cb513550-94cd-4aa9-b207-87c213e801e5" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425611    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb513550-94cd-4aa9-b207-87c213e801e5" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425625    5817 cpu_manager.go:414] "containerMap: removing container" podUID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425631    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425641    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5237060b-3f82-4aea-a86b-a39893e57014" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425647    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5237060b-3f82-4aea-a86b-a39893e57014" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425656    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425663    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerName="pull"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425673    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425681    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerName="util"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425757    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc65d628-9b6b-43ec-83cd-b0d0fec15195" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425771    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1bc56f3-1075-4ac7-9675-c8be38dc1054" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425781    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2c724a-3cb2-4b7d-961f-b9e51a1dcf17" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425789    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bb3d79-e89a-4295-b089-afb06b5fd6d5" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425797    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eff77a4-3c4c-4b86-a76e-875e683631bd" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425806    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb513550-94cd-4aa9-b207-87c213e801e5" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425814    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5660ccd9-123d-4de8-9d98-da67d0f0bafe" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425821    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df10d9d-8f18-41d9-9cc4-0a5bf45e256c" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425828    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="564b0ac9-0744-4cf0-97cc-f67c9c23e124" containerName="extract"
Nov 03 10:34:24.426051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.425836    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5237060b-3f82-4aea-a86b-a39893e57014" containerName="extract"
Nov 03 10:34:24.428815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.428794    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"
Nov 03 10:34:24.436100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.435913    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:34:24.436100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.435912    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 10:34:24.436308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.436288    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-lm5tz\""
Nov 03 10:34:24.437539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.437516    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"]
Nov 03 10:34:24.575016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.574991    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8585ea64-14f8-49ce-92df-94b5421640ce-tmp\") pod \"jobset-operator-674cd59dc5-l2h46\" (UID: \"8585ea64-14f8-49ce-92df-94b5421640ce\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"
Nov 03 10:34:24.575131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.575023    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqff9\" (UniqueName: \"kubernetes.io/projected/8585ea64-14f8-49ce-92df-94b5421640ce-kube-api-access-pqff9\") pod \"jobset-operator-674cd59dc5-l2h46\" (UID: \"8585ea64-14f8-49ce-92df-94b5421640ce\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"
Nov 03 10:34:24.675824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.675754    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8585ea64-14f8-49ce-92df-94b5421640ce-tmp\") pod \"jobset-operator-674cd59dc5-l2h46\" (UID: \"8585ea64-14f8-49ce-92df-94b5421640ce\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"
Nov 03 10:34:24.675824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.675796    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqff9\" (UniqueName: \"kubernetes.io/projected/8585ea64-14f8-49ce-92df-94b5421640ce-kube-api-access-pqff9\") pod \"jobset-operator-674cd59dc5-l2h46\" (UID: \"8585ea64-14f8-49ce-92df-94b5421640ce\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"
Nov 03 10:34:24.676124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.676108    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8585ea64-14f8-49ce-92df-94b5421640ce-tmp\") pod \"jobset-operator-674cd59dc5-l2h46\" (UID: \"8585ea64-14f8-49ce-92df-94b5421640ce\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"
Nov 03 10:34:24.683958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.683938    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqff9\" (UniqueName: \"kubernetes.io/projected/8585ea64-14f8-49ce-92df-94b5421640ce-kube-api-access-pqff9\") pod \"jobset-operator-674cd59dc5-l2h46\" (UID: \"8585ea64-14f8-49ce-92df-94b5421640ce\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"
Nov 03 10:34:24.748479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:24.748457    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"
Nov 03 10:34:25.017340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:25.017242    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46"]
Nov 03 10:34:25.020068 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:34:25.020040    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8585ea64_14f8_49ce_92df_94b5421640ce.slice/crio-332d0eed10185ba86c0f19f554395e3ff7f2d71d1517be924a61495ca9a0fa3a WatchSource:0}: Error finding container 332d0eed10185ba86c0f19f554395e3ff7f2d71d1517be924a61495ca9a0fa3a: Status 404 returned error can't find the container with id 332d0eed10185ba86c0f19f554395e3ff7f2d71d1517be924a61495ca9a0fa3a
Nov 03 10:34:25.382736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:25.382650    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46" event={"ID":"8585ea64-14f8-49ce-92df-94b5421640ce","Type":"ContainerStarted","Data":"332d0eed10185ba86c0f19f554395e3ff7f2d71d1517be924a61495ca9a0fa3a"}
Nov 03 10:34:27.393312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:27.393278    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46" event={"ID":"8585ea64-14f8-49ce-92df-94b5421640ce","Type":"ContainerStarted","Data":"bce984ae3b146532acd4579d6ae25dbafb74fa0fe889d6ae767c095cafecb6eb"}
Nov 03 10:34:27.409394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:27.409310    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l2h46" podStartSLOduration=1.28535212 podStartE2EDuration="3.409294424s" totalImagesPullingTime="2.123942304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:27.407997679 +0000 UTC m=+297.613579732" watchObservedRunningTime="2026-11-03 10:34:27.409294424 +0000 UTC m=+297.614876512"
Nov 03 10:34:27.662222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:27.662140    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:28.854433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:28.854401    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:30.969957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:30.969895    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:31.672189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:31.672155    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:31.824565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:31.824526    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:33.011338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.011300    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:33.098103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.098065    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"]
Nov 03 10:34:33.101387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.101364    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.127517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.127487    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"]
Nov 03 10:34:33.188473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.188445    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 10:34:33.188764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.188747    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 10:34:33.188841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.188768    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 10:34:33.188841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.188772    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 10:34:33.188841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.188798    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 10:34:33.188841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.188836    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-s78mx\""
Nov 03 10:34:33.247308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.247272    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.247458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.247316    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.247458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.247393    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmlgc\" (UniqueName: \"kubernetes.io/projected/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-kube-api-access-cmlgc\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.247458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.247427    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.247555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.247518    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.247619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.247559    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.265724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.265647    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:33.348648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.348614    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.348817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.348659    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmlgc\" (UniqueName: \"kubernetes.io/projected/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-kube-api-access-cmlgc\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.348817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.348705    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.348817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.348742    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.348817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.348782    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.349029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.348821    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.349175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.349157    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.351104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.351079    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.351407 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.351385    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.351475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.351432    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.351519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.351479    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.357077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.357059    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmlgc\" (UniqueName: \"kubernetes.io/projected/08d3baa2-4f3b-4653-ab09-6cd67ce3d72c-kube-api-access-cmlgc\") pod \"opendatahub-operator-controller-manager-6688c64986-lb282\" (UID: \"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.413595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.413541    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:33.432942 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.432917    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:33.570961 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.570881    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:33.716855 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.716824    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"]
Nov 03 10:34:33.721432 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:34:33.721407    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d3baa2_4f3b_4653_ab09_6cd67ce3d72c.slice/crio-bf3610cadc785474bf2cda4a39cc93f66ef9283cf81f83e25d1879bca2f584f5 WatchSource:0}: Error finding container bf3610cadc785474bf2cda4a39cc93f66ef9283cf81f83e25d1879bca2f584f5: Status 404 returned error can't find the container with id bf3610cadc785474bf2cda4a39cc93f66ef9283cf81f83e25d1879bca2f584f5
Nov 03 10:34:33.743844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:33.743815    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:34.428999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:34.424657    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282" event={"ID":"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c","Type":"ContainerStarted","Data":"bf3610cadc785474bf2cda4a39cc93f66ef9283cf81f83e25d1879bca2f584f5"}
Nov 03 10:34:34.653630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:34.653596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:35.503993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:35.503956    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:36.072260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:36.072222    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:36.782174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:36.782137    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:37.292926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:37.292892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:37.440623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:37.440597    5817 generic.go:356] "Generic (PLEG): container finished" podID="08d3baa2-4f3b-4653-ab09-6cd67ce3d72c" containerID="93f0f6ac15276cc95128f665a0300bb43e0170d93f1f81d38e5e17ad33f202fd" exitCode=0
Nov 03 10:34:37.440747 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:37.440665    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282" event={"ID":"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c","Type":"ContainerDied","Data":"93f0f6ac15276cc95128f665a0300bb43e0170d93f1f81d38e5e17ad33f202fd"}
Nov 03 10:34:37.446247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:37.446207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:38.447367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:38.447334    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282" event={"ID":"08d3baa2-4f3b-4653-ab09-6cd67ce3d72c","Type":"ContainerStarted","Data":"e9113c4718a49b2bad901a4f7365e864449067ee058d35a9ec0aa6f78228e58e"}
Nov 03 10:34:38.447756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:38.447456    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:38.483932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:38.483888    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282" podStartSLOduration=1.479207025 podStartE2EDuration="5.483875537s" totalImagesPullingTime="3.542506178s" totalInitContainerRuntime="462.162334ms" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:38.482401988 +0000 UTC m=+308.687984039" watchObservedRunningTime="2026-11-03 10:34:38.483875537 +0000 UTC m=+308.689457587"
Nov 03 10:34:40.706891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:40.706854    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:40.852310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:40.852266    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:41.305737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:41.305701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:42.865862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:42.865833    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:43.097038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:43.096998    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:45.209448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.209410    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"]
Nov 03 10:34:45.214753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.214732    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"
Nov 03 10:34:45.220917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.220758    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 10:34:45.220917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.220844    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:34:45.222143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.222122    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-z5bws\""
Nov 03 10:34:45.223900 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.223876    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"]
Nov 03 10:34:45.343305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.343281    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fedbf1e-ef0d-4989-9499-dac74d3584fb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j\" (UID: \"6fedbf1e-ef0d-4989-9499-dac74d3584fb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"
Nov 03 10:34:45.343433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.343314    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fn9\" (UniqueName: \"kubernetes.io/projected/6fedbf1e-ef0d-4989-9499-dac74d3584fb-kube-api-access-66fn9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j\" (UID: \"6fedbf1e-ef0d-4989-9499-dac74d3584fb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"
Nov 03 10:34:45.444217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.444191    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fedbf1e-ef0d-4989-9499-dac74d3584fb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j\" (UID: \"6fedbf1e-ef0d-4989-9499-dac74d3584fb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"
Nov 03 10:34:45.444344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.444224    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fn9\" (UniqueName: \"kubernetes.io/projected/6fedbf1e-ef0d-4989-9499-dac74d3584fb-kube-api-access-66fn9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j\" (UID: \"6fedbf1e-ef0d-4989-9499-dac74d3584fb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"
Nov 03 10:34:45.444594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.444564    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fedbf1e-ef0d-4989-9499-dac74d3584fb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j\" (UID: \"6fedbf1e-ef0d-4989-9499-dac74d3584fb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"
Nov 03 10:34:45.452401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.452384    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fn9\" (UniqueName: \"kubernetes.io/projected/6fedbf1e-ef0d-4989-9499-dac74d3584fb-kube-api-access-66fn9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j\" (UID: \"6fedbf1e-ef0d-4989-9499-dac74d3584fb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"
Nov 03 10:34:45.527266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.527213    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"
Nov 03 10:34:45.809997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:45.809906    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j"]
Nov 03 10:34:45.812685 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:34:45.812653    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fedbf1e_ef0d_4989_9499_dac74d3584fb.slice/crio-05a11ce8fe55fce1099ecdc8318cc25a4b61d6ee75f4e1f18d56ea46802e4d2e WatchSource:0}: Error finding container 05a11ce8fe55fce1099ecdc8318cc25a4b61d6ee75f4e1f18d56ea46802e4d2e: Status 404 returned error can't find the container with id 05a11ce8fe55fce1099ecdc8318cc25a4b61d6ee75f4e1f18d56ea46802e4d2e
Nov 03 10:34:46.054546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.049940    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"]
Nov 03 10:34:46.056960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.056939    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"
Nov 03 10:34:46.065273 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.065248    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 10:34:46.065677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.065329    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 10:34:46.065677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.065354    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6xd4q\""
Nov 03 10:34:46.065677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.065442    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 10:34:46.067998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.067967    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"]
Nov 03 10:34:46.149761 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.149728    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjmw5\" (UniqueName: \"kubernetes.io/projected/cf5191a6-ee98-4b6b-b267-2af0e92b6f40-kube-api-access-qjmw5\") pod \"dns-operator-controller-manager-8557cbdb84-vrn6v\" (UID: \"cf5191a6-ee98-4b6b-b267-2af0e92b6f40\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"
Nov 03 10:34:46.250877 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.250847    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmw5\" (UniqueName: \"kubernetes.io/projected/cf5191a6-ee98-4b6b-b267-2af0e92b6f40-kube-api-access-qjmw5\") pod \"dns-operator-controller-manager-8557cbdb84-vrn6v\" (UID: \"cf5191a6-ee98-4b6b-b267-2af0e92b6f40\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"
Nov 03 10:34:46.260057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.260037    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmw5\" (UniqueName: \"kubernetes.io/projected/cf5191a6-ee98-4b6b-b267-2af0e92b6f40-kube-api-access-qjmw5\") pod \"dns-operator-controller-manager-8557cbdb84-vrn6v\" (UID: \"cf5191a6-ee98-4b6b-b267-2af0e92b6f40\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"
Nov 03 10:34:46.370527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.370463    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"
Nov 03 10:34:46.483719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.483691    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j" event={"ID":"6fedbf1e-ef0d-4989-9499-dac74d3584fb","Type":"ContainerStarted","Data":"05a11ce8fe55fce1099ecdc8318cc25a4b61d6ee75f4e1f18d56ea46802e4d2e"}
Nov 03 10:34:46.578933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.578901    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:46.657991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:46.657726    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"]
Nov 03 10:34:46.660881 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:34:46.660850    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5191a6_ee98_4b6b_b267_2af0e92b6f40.slice/crio-f6b2c15eb8c737e62c99c754efbb3bdbede92b462a0a10541bac64219482af17 WatchSource:0}: Error finding container f6b2c15eb8c737e62c99c754efbb3bdbede92b462a0a10541bac64219482af17: Status 404 returned error can't find the container with id f6b2c15eb8c737e62c99c754efbb3bdbede92b462a0a10541bac64219482af17
Nov 03 10:34:47.075730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:47.075690    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:47.493653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:47.493617    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v" event={"ID":"cf5191a6-ee98-4b6b-b267-2af0e92b6f40","Type":"ContainerStarted","Data":"f6b2c15eb8c737e62c99c754efbb3bdbede92b462a0a10541bac64219482af17"}
Nov 03 10:34:47.592747 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:47.592713    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:48.502590 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:48.502545    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j" event={"ID":"6fedbf1e-ef0d-4989-9499-dac74d3584fb","Type":"ContainerStarted","Data":"214d91e98d41ee532dea6b83f412bbcf8fb9bdb93b6668bf49df8d90467f2415"}
Nov 03 10:34:48.526174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:48.524414    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tnc4j" podStartSLOduration=1.490473346 podStartE2EDuration="3.524392933s" totalImagesPullingTime="2.033919587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:48.523673846 +0000 UTC m=+318.729255892" watchObservedRunningTime="2026-11-03 10:34:48.524392933 +0000 UTC m=+318.729974985"
Nov 03 10:34:49.157331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.157298    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"]
Nov 03 10:34:49.160838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.160819    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.166815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.166692    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 10:34:49.166815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.166761    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xpkj5\""
Nov 03 10:34:49.167941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.167918    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"]
Nov 03 10:34:49.213608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.213563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:49.256061 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.256034    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 10:34:49.279432 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.279398    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5049482-53a8-4070-8b53-47a3ce9cf145-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.279602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.279527    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d5049482-53a8-4070-8b53-47a3ce9cf145-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.279698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.279682    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgq9h\" (UniqueName: \"kubernetes.io/projected/d5049482-53a8-4070-8b53-47a3ce9cf145-kube-api-access-sgq9h\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.380817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.380776    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgq9h\" (UniqueName: \"kubernetes.io/projected/d5049482-53a8-4070-8b53-47a3ce9cf145-kube-api-access-sgq9h\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.380817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.380819    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5049482-53a8-4070-8b53-47a3ce9cf145-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.381053 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.380870    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d5049482-53a8-4070-8b53-47a3ce9cf145-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.381631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.381610    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d5049482-53a8-4070-8b53-47a3ce9cf145-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.383306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.383287    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5049482-53a8-4070-8b53-47a3ce9cf145-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.394186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.394160    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgq9h\" (UniqueName: \"kubernetes.io/projected/d5049482-53a8-4070-8b53-47a3ce9cf145-kube-api-access-sgq9h\") pod \"kuadrant-console-plugin-66c4fb575d-jjvlt\" (UID: \"d5049482-53a8-4070-8b53-47a3ce9cf145\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.422280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.422219    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:49.454800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.454779    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-lb282"
Nov 03 10:34:49.475008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.474990    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"
Nov 03 10:34:49.769183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.769160    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt"]
Nov 03 10:34:49.771285 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:34:49.771260    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5049482_53a8_4070_8b53_47a3ce9cf145.slice/crio-64f305ac703bf66137b4913bb1e4136001c4cbfb3d9fc7b5e676cb91bcdbdbc5 WatchSource:0}: Error finding container 64f305ac703bf66137b4913bb1e4136001c4cbfb3d9fc7b5e676cb91bcdbdbc5: Status 404 returned error can't find the container with id 64f305ac703bf66137b4913bb1e4136001c4cbfb3d9fc7b5e676cb91bcdbdbc5
Nov 03 10:34:49.810136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:49.810099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:50.027035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.026947    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:50.514162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.514122    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v" event={"ID":"cf5191a6-ee98-4b6b-b267-2af0e92b6f40","Type":"ContainerStarted","Data":"4b024276ef71cd33a369833a43101bab8a4c3d8abbe81df1273d0e663bbdc651"}
Nov 03 10:34:50.514337 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.514267    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"
Nov 03 10:34:50.515686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.515654    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt" event={"ID":"d5049482-53a8-4070-8b53-47a3ce9cf145","Type":"ContainerStarted","Data":"64f305ac703bf66137b4913bb1e4136001c4cbfb3d9fc7b5e676cb91bcdbdbc5"}
Nov 03 10:34:50.542258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.542217    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v" podStartSLOduration=1.7483512129999998 podStartE2EDuration="4.542205088s" totalImagesPullingTime="2.793853875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:50.541309937 +0000 UTC m=+320.746891990" watchObservedRunningTime="2026-11-03 10:34:50.542205088 +0000 UTC m=+320.747787139"
Nov 03 10:34:50.827954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.827872    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"]
Nov 03 10:34:50.831843 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.831816    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:50.842537 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.841872    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"]
Nov 03 10:34:50.847596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.847277    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 10:34:50.847596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.847359    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 10:34:50.864005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.863971    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-f4zqh\""
Nov 03 10:34:50.999232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.999185    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvhs\" (UniqueName: \"kubernetes.io/projected/e2fb455f-812f-4892-9abf-da9bbc072042-kube-api-access-sfvhs\") pod \"cert-manager-webhook-5dbbbf75cc-q6sw7\" (UID: \"e2fb455f-812f-4892-9abf-da9bbc072042\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:50.999387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:50.999249    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2fb455f-812f-4892-9abf-da9bbc072042-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-q6sw7\" (UID: \"e2fb455f-812f-4892-9abf-da9bbc072042\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:51.100312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.100225    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvhs\" (UniqueName: \"kubernetes.io/projected/e2fb455f-812f-4892-9abf-da9bbc072042-kube-api-access-sfvhs\") pod \"cert-manager-webhook-5dbbbf75cc-q6sw7\" (UID: \"e2fb455f-812f-4892-9abf-da9bbc072042\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:51.100312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.100288    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2fb455f-812f-4892-9abf-da9bbc072042-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-q6sw7\" (UID: \"e2fb455f-812f-4892-9abf-da9bbc072042\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:51.108859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.108834    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvhs\" (UniqueName: \"kubernetes.io/projected/e2fb455f-812f-4892-9abf-da9bbc072042-kube-api-access-sfvhs\") pod \"cert-manager-webhook-5dbbbf75cc-q6sw7\" (UID: \"e2fb455f-812f-4892-9abf-da9bbc072042\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:51.112159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.112129    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2fb455f-812f-4892-9abf-da9bbc072042-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-q6sw7\" (UID: \"e2fb455f-812f-4892-9abf-da9bbc072042\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:51.154159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.154110    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:51.440147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.440123    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"]
Nov 03 10:34:51.442583 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:34:51.442496    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2fb455f_812f_4892_9abf_da9bbc072042.slice/crio-5f375b02d3cc71a6da01ba39f6ea93f3b667961432f03b35b8fb692bfff04636 WatchSource:0}: Error finding container 5f375b02d3cc71a6da01ba39f6ea93f3b667961432f03b35b8fb692bfff04636: Status 404 returned error can't find the container with id 5f375b02d3cc71a6da01ba39f6ea93f3b667961432f03b35b8fb692bfff04636
Nov 03 10:34:51.470143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.470093    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:51.521965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.521933    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7" event={"ID":"e2fb455f-812f-4892-9abf-da9bbc072042","Type":"ContainerStarted","Data":"5f375b02d3cc71a6da01ba39f6ea93f3b667961432f03b35b8fb692bfff04636"}
Nov 03 10:34:51.607135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:51.607107    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:52.024196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:52.024154    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:52.176644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:52.176609    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:52.423844 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:34:52.423769    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:34:52.471295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:52.471252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:53.321968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:53.321912    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:55.544809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:55.544774    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt" event={"ID":"d5049482-53a8-4070-8b53-47a3ce9cf145","Type":"ContainerStarted","Data":"143622b628fac910b0ddf7fe3222fafbfc47176de8341cb06476f16440954621"}
Nov 03 10:34:55.546378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:55.546354    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7" event={"ID":"e2fb455f-812f-4892-9abf-da9bbc072042","Type":"ContainerStarted","Data":"c1e6d632ea323367caf75bb198c6649aadeb984d6432493e37b17bd636ecc225"}
Nov 03 10:34:55.546486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:55.546429    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:34:55.561137 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:55.561092    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jjvlt" podStartSLOduration=1.64301991 podStartE2EDuration="6.561078548s" totalImagesPullingTime="4.918058638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:55.560367703 +0000 UTC m=+325.765949764" watchObservedRunningTime="2026-11-03 10:34:55.561078548 +0000 UTC m=+325.766660599"
Nov 03 10:34:55.578613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:55.578557    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7" podStartSLOduration=1.926751474 podStartE2EDuration="5.57854585s" totalImagesPullingTime="3.651794376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:55.577267009 +0000 UTC m=+325.782849062" watchObservedRunningTime="2026-11-03 10:34:55.57854585 +0000 UTC m=+325.784127901"
Nov 03 10:34:56.919699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:56.919659    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:57.076280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:57.076246    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:57.865719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:57.865684    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:58.076377 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.076344    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:58.578681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.578650    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"]
Nov 03 10:34:58.582001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.581986    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:34:58.590367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.590348    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cxm7r\""
Nov 03 10:34:58.595230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.595206    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"]
Nov 03 10:34:58.671913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.671878    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7445\" (UniqueName: \"kubernetes.io/projected/63222367-1778-4565-9e4b-c2b33621751e-kube-api-access-t7445\") pod \"kuadrant-operator-controller-manager-678d87766f-klrdr\" (UID: \"63222367-1778-4565-9e4b-c2b33621751e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:34:58.672287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.672031    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63222367-1778-4565-9e4b-c2b33621751e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-klrdr\" (UID: \"63222367-1778-4565-9e4b-c2b33621751e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:34:58.773243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.773212    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63222367-1778-4565-9e4b-c2b33621751e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-klrdr\" (UID: \"63222367-1778-4565-9e4b-c2b33621751e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:34:58.773243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.773254    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7445\" (UniqueName: \"kubernetes.io/projected/63222367-1778-4565-9e4b-c2b33621751e-kube-api-access-t7445\") pod \"kuadrant-operator-controller-manager-678d87766f-klrdr\" (UID: \"63222367-1778-4565-9e4b-c2b33621751e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:34:58.773610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.773563    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63222367-1778-4565-9e4b-c2b33621751e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-klrdr\" (UID: \"63222367-1778-4565-9e4b-c2b33621751e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:34:58.785560 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.785532    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7445\" (UniqueName: \"kubernetes.io/projected/63222367-1778-4565-9e4b-c2b33621751e-kube-api-access-t7445\") pod \"kuadrant-operator-controller-manager-678d87766f-klrdr\" (UID: \"63222367-1778-4565-9e4b-c2b33621751e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:34:58.870924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.870832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:58.894400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:58.894372    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:34:59.011258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:59.011059    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:59.177659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:59.177584    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"]
Nov 03 10:34:59.181156 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:34:59.181129    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63222367_1778_4565_9e4b_c2b33621751e.slice/crio-921bfffdc7167bd6b7c19d1ee4c2c72eded8481217f42ca8904f67c42733eb9f WatchSource:0}: Error finding container 921bfffdc7167bd6b7c19d1ee4c2c72eded8481217f42ca8904f67c42733eb9f: Status 404 returned error can't find the container with id 921bfffdc7167bd6b7c19d1ee4c2c72eded8481217f42ca8904f67c42733eb9f
Nov 03 10:34:59.278139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:59.278109    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:59.425448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:59.425407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:34:59.566715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:34:59.566683    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr" event={"ID":"63222367-1778-4565-9e4b-c2b33621751e","Type":"ContainerStarted","Data":"921bfffdc7167bd6b7c19d1ee4c2c72eded8481217f42ca8904f67c42733eb9f"}
Nov 03 10:35:00.002242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:00.002207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:01.525004 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:01.524979    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vrn6v"
Nov 03 10:35:01.553306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:01.553278    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q6sw7"
Nov 03 10:35:04.595550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.595514    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr" event={"ID":"63222367-1778-4565-9e4b-c2b33621751e","Type":"ContainerStarted","Data":"f5835271d1de6c944f79e95c3c9a27c011a7c4c7fcd21881b245e6ab087eb45e"}
Nov 03 10:35:04.596067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.595910    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:35:04.623492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.623438    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr" podStartSLOduration=2.105305141 podStartE2EDuration="6.623424848s" totalImagesPullingTime="4.518119707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:04.622171956 +0000 UTC m=+334.827754006" watchObservedRunningTime="2026-11-03 10:35:04.623424848 +0000 UTC m=+334.829006889"
Nov 03 10:35:04.696128 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.696077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:04.882816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.882720    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:04.972807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.972770    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"]
Nov 03 10:35:04.977368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.977345    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:04.989345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.988348    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 10:35:04.989345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.988460    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5xvj2\""
Nov 03 10:35:04.989345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.988519    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 10:35:04.991339 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:04.991315    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"]
Nov 03 10:35:05.031015 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.030987    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9zsc\" (UniqueName: \"kubernetes.io/projected/86d8e220-c327-4e65-95d6-a7394f5c1b9e-kube-api-access-z9zsc\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.031210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.031026    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86d8e220-c327-4e65-95d6-a7394f5c1b9e-tmp\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.031210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.031141    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d8e220-c327-4e65-95d6-a7394f5c1b9e-tls-certs\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.074305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.074270    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:05.116748 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.116714    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"]
Nov 03 10:35:05.120231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.120210    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.129096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.129072    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-489dz\""
Nov 03 10:35:05.131550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.131524    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9zsc\" (UniqueName: \"kubernetes.io/projected/86d8e220-c327-4e65-95d6-a7394f5c1b9e-kube-api-access-z9zsc\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.131660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.131584    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.131660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.131620    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.131660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.131648    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86d8e220-c327-4e65-95d6-a7394f5c1b9e-tmp\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.132044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.131677    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.132044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.131704    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm5x4\" (UniqueName: \"kubernetes.io/projected/282aee57-34ba-4c1d-a305-13b93667a0cc-kube-api-access-gm5x4\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.132044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.131739    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.132360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.132341    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d8e220-c327-4e65-95d6-a7394f5c1b9e-tls-certs\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.132437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.132390    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.132437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.132430    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282aee57-34ba-4c1d-a305-13b93667a0cc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.132520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.132470    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.132520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.132501    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.134406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.134352    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86d8e220-c327-4e65-95d6-a7394f5c1b9e-tmp\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.135381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.135356    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d8e220-c327-4e65-95d6-a7394f5c1b9e-tls-certs\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.139891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.139874    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"]
Nov 03 10:35:05.153919 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.153890    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9zsc\" (UniqueName: \"kubernetes.io/projected/86d8e220-c327-4e65-95d6-a7394f5c1b9e-kube-api-access-z9zsc\") pod \"kube-auth-proxy-5cb777df86-9cpw5\" (UID: \"86d8e220-c327-4e65-95d6-a7394f5c1b9e\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.233415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233379    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.233640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233436    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.233640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233468    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.233640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233497    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.233640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233521    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5x4\" (UniqueName: \"kubernetes.io/projected/282aee57-34ba-4c1d-a305-13b93667a0cc-kube-api-access-gm5x4\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.233640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233553    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.233920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233654    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.233920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233697    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282aee57-34ba-4c1d-a305-13b93667a0cc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.233920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233737    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.234069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.233968    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.234069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.234050    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.234290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.234261    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.234403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.234369    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.234780 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.234757    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282aee57-34ba-4c1d-a305-13b93667a0cc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.236413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.236393    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.236512 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.236492    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.248018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.247985    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282aee57-34ba-4c1d-a305-13b93667a0cc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.248286 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.248269    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5x4\" (UniqueName: \"kubernetes.io/projected/282aee57-34ba-4c1d-a305-13b93667a0cc-kube-api-access-gm5x4\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fn6gpr\" (UID: \"282aee57-34ba-4c1d-a305-13b93667a0cc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.252474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.252447    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:05.291738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.291720    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"
Nov 03 10:35:05.385724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.385646    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:05.437305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.437273    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:05.531764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.531731    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:05.582781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.582741    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5"]
Nov 03 10:35:05.589394 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:35:05.589363    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d8e220_c327_4e65_95d6_a7394f5c1b9e.slice/crio-1413697eb5499c58cc597a5a3d3ec69b65e22f8eea5bd06af01e1a37d5ccd715 WatchSource:0}: Error finding container 1413697eb5499c58cc597a5a3d3ec69b65e22f8eea5bd06af01e1a37d5ccd715: Status 404 returned error can't find the container with id 1413697eb5499c58cc597a5a3d3ec69b65e22f8eea5bd06af01e1a37d5ccd715
Nov 03 10:35:05.604291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.604235    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5" event={"ID":"86d8e220-c327-4e65-95d6-a7394f5c1b9e","Type":"ContainerStarted","Data":"1413697eb5499c58cc597a5a3d3ec69b65e22f8eea5bd06af01e1a37d5ccd715"}
Nov 03 10:35:05.721466 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.721433    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"]
Nov 03 10:35:05.723962 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:35:05.723939    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282aee57_34ba_4c1d_a305_13b93667a0cc.slice/crio-3ea6c0fc9b2ecc80cf29f78e889bcd33a9947cc7002b66d7de5e813601cb9fb5 WatchSource:0}: Error finding container 3ea6c0fc9b2ecc80cf29f78e889bcd33a9947cc7002b66d7de5e813601cb9fb5: Status 404 returned error can't find the container with id 3ea6c0fc9b2ecc80cf29f78e889bcd33a9947cc7002b66d7de5e813601cb9fb5
Nov 03 10:35:05.726163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.726133    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:35:05.726248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.726195    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:35:05.726248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:05.726223    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:35:06.341810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:06.341708    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:06.481786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:06.481745    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:06.611968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:06.611877    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr" event={"ID":"282aee57-34ba-4c1d-a305-13b93667a0cc","Type":"ContainerStarted","Data":"edbf7f248f9f45269316b544992e2a4aea7801e058167ae9e12505bf4db064e5"}
Nov 03 10:35:06.612435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:06.611990    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr" event={"ID":"282aee57-34ba-4c1d-a305-13b93667a0cc","Type":"ContainerStarted","Data":"3ea6c0fc9b2ecc80cf29f78e889bcd33a9947cc7002b66d7de5e813601cb9fb5"}
Nov 03 10:35:06.637317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:06.637254    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr" podStartSLOduration=1.6372333239999999 podStartE2EDuration="1.637233324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:06.6348661 +0000 UTC m=+336.840448163" watchObservedRunningTime="2026-11-03 10:35:06.637233324 +0000 UTC m=+336.842815376"
Nov 03 10:35:06.663119 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:06.663087    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:07.046960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:07.046913    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:07.438087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:07.438051    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:07.443841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:07.443812    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:07.616839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:07.616812    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:07.617924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:07.617905    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fn6gpr"
Nov 03 10:35:08.622987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:08.622958    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5" event={"ID":"86d8e220-c327-4e65-95d6-a7394f5c1b9e","Type":"ContainerStarted","Data":"cdf005b0b0dcc048f1cd35fda168a82a102a31a55422443f9ffb6bec1a7a5cc2"}
Nov 03 10:35:08.639819 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:08.639778    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cb777df86-9cpw5" podStartSLOduration=2.158278166 podStartE2EDuration="4.639763414s" totalImagesPullingTime="2.481485248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:08.637842979 +0000 UTC m=+338.843425030" watchObservedRunningTime="2026-11-03 10:35:08.639763414 +0000 UTC m=+338.845345462"
Nov 03 10:35:09.873381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:09.873326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:10.000665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:10.000633    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:13.664718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:13.664684    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:13.797712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:13.797672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:13.929210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:13.929183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:15.606723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:15.606690    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-klrdr"
Nov 03 10:35:17.556117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:17.556078    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:17.744310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:17.744277    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:22.410451 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:35:22.410409    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:35:24.026538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:24.026505    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:24.161697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:24.161664    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:25.241963 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:25.241933    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:25.417938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:25.417906    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:28.741506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:28.741474    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:28.873890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:28.873858    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:30.943246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:30.943210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:32.546232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:32.546195    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:32.673520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:32.673491    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:33.149837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:33.149796    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:33.311662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:33.311627    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:35.114392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.114358    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:35.346138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.346108    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:35:35.366324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.366259    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:35:35.366460 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.366353    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:35.369605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.369564    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 10:35:35.377952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.377930    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:35:35.507542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.507515    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b444367-e539-4961-a151-0847a752f0b1-config-file\") pod \"limitador-limitador-7c9b87d574-82pqb\" (UID: \"6b444367-e539-4961-a151-0847a752f0b1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:35.507722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.507611    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbdg\" (UniqueName: \"kubernetes.io/projected/6b444367-e539-4961-a151-0847a752f0b1-kube-api-access-8wbdg\") pod \"limitador-limitador-7c9b87d574-82pqb\" (UID: \"6b444367-e539-4961-a151-0847a752f0b1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:35.608541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.608512    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b444367-e539-4961-a151-0847a752f0b1-config-file\") pod \"limitador-limitador-7c9b87d574-82pqb\" (UID: \"6b444367-e539-4961-a151-0847a752f0b1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:35.608697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.608630    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbdg\" (UniqueName: \"kubernetes.io/projected/6b444367-e539-4961-a151-0847a752f0b1-kube-api-access-8wbdg\") pod \"limitador-limitador-7c9b87d574-82pqb\" (UID: \"6b444367-e539-4961-a151-0847a752f0b1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:35.609104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.609084    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b444367-e539-4961-a151-0847a752f0b1-config-file\") pod \"limitador-limitador-7c9b87d574-82pqb\" (UID: \"6b444367-e539-4961-a151-0847a752f0b1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:35.616197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.616175    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbdg\" (UniqueName: \"kubernetes.io/projected/6b444367-e539-4961-a151-0847a752f0b1-kube-api-access-8wbdg\") pod \"limitador-limitador-7c9b87d574-82pqb\" (UID: \"6b444367-e539-4961-a151-0847a752f0b1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:35.679674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.679657    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:35.947936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:35.947915    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:35:35.949864 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:35:35.949827    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b444367_e539_4961_a151_0847a752f0b1.slice/crio-83500be4f39ca89161091e30d062162aff8cbbf19852f1d4868ad129087ab313 WatchSource:0}: Error finding container 83500be4f39ca89161091e30d062162aff8cbbf19852f1d4868ad129087ab313: Status 404 returned error can't find the container with id 83500be4f39ca89161091e30d062162aff8cbbf19852f1d4868ad129087ab313
Nov 03 10:35:36.758825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:36.758797    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb" event={"ID":"6b444367-e539-4961-a151-0847a752f0b1","Type":"ContainerStarted","Data":"83500be4f39ca89161091e30d062162aff8cbbf19852f1d4868ad129087ab313"}
Nov 03 10:35:37.584943 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:37.584855    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:37.717025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:37.716995    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:37.765467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:37.765438    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb" event={"ID":"6b444367-e539-4961-a151-0847a752f0b1","Type":"ContainerStarted","Data":"ed4c623b9945c03e1c7a6597ee312f8ae0af008f9b4c1fedae352219cec594e5"}
Nov 03 10:35:37.765817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:37.765589    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:37.781979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:37.781934    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb" podStartSLOduration=1.526287479 podStartE2EDuration="2.781353961s" totalImagesPullingTime="1.255066482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:37.78081628 +0000 UTC m=+367.986398332" watchObservedRunningTime="2026-11-03 10:35:37.781353961 +0000 UTC m=+367.986936013"
Nov 03 10:35:37.953625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:37.953598    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:38.085025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:38.084991    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:38.350319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:38.350234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:38.486258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:38.486227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:43.799548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:43.799475    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:43.933774 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:43.933740    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:44.423598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:44.423540    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:44.550965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:44.550929    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:48.562632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:48.562595    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:48.700198 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:48.700167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:48.771043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:48.771023    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-82pqb"
Nov 03 10:35:52.410410 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:35:52.410376    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:35:53.133088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:53.133049    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:53.311915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:53.311892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:35:54.006474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:35:54.006434    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:01.064297 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:01.064264    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:01.224558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:01.224524    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:01.372631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:01.372527    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:01.544162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:01.544131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:02.186940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:02.186904    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:02.333669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:02.333635    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:02.487137 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:02.487051    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:13.018326 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:13.018289    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:13.166984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:13.166949    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:13.302254 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:13.302191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:17.824750 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:17.824715    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:18.023521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:18.023485    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:20.127545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:20.127509    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:20.614897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:20.614860    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:22.410845 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:36:22.410803    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:36:22.810038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:22.810004    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:26.750510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:26.750481    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:26.880907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:26.880872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:30.984542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:30.984508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:48.870040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:48.869983    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:49.014322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:49.014294    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:50.188742 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:50.188708    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:50.327441 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:50.327403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:36:51.033667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:36:51.033633    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:36:52.425424 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:36:52.425391    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:37:01.170276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:01.170244    5817 generic.go:356] "Generic (PLEG): container finished" podID="edf3ef4f-1840-4e68-9049-ec8a195d5e9b" containerID="32fdb7a299da075ba3fc1005c70a500fabd01e71fb8243fc9d2a32d1734bd6e9" exitCode=0
Nov 03 10:37:01.170722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:01.170292    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" event={"ID":"edf3ef4f-1840-4e68-9049-ec8a195d5e9b","Type":"ContainerDied","Data":"32fdb7a299da075ba3fc1005c70a500fabd01e71fb8243fc9d2a32d1734bd6e9"}
Nov 03 10:37:01.170722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:01.170670    5817 scope.go:122] "RemoveContainer" containerID="32fdb7a299da075ba3fc1005c70a500fabd01e71fb8243fc9d2a32d1734bd6e9"
Nov 03 10:37:01.171777 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:01.171749    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:37:01.171856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:01.171842    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:37:01.171898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:01.171886    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:37:01.613217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:01.613181    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:37:01.744064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:01.744023    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:37:02.178064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:02.178028    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" event={"ID":"edf3ef4f-1840-4e68-9049-ec8a195d5e9b","Type":"ContainerStarted","Data":"b16c9a98a9bb315ffff5c529b773eb45759e9e2c4816d613bcaa7a2484794806"}
Nov 03 10:37:02.181538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:02.181502    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:37:02.181773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:02.181735    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-c7n4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 10:37:02.181905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:02.181803    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" podUID="edf3ef4f-1840-4e68-9049-ec8a195d5e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 10:37:03.181729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:03.181694    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-c7n4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 10:37:03.182218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:03.181741    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" podUID="edf3ef4f-1840-4e68-9049-ec8a195d5e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 10:37:04.181944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:04.181906    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-c7n4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 10:37:04.182344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:04.181971    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" podUID="edf3ef4f-1840-4e68-9049-ec8a195d5e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 10:37:05.181856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:05.181820    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-c7n4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 10:37:05.182055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:05.181880    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" podUID="edf3ef4f-1840-4e68-9049-ec8a195d5e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 10:37:06.181993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:06.181953    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-c7n4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Nov 03 10:37:06.182195 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:06.182020    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" podUID="edf3ef4f-1840-4e68-9049-ec8a195d5e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Nov 03 10:37:07.185416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:07.185383    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:37:07.185828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:07.185793    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:37:07.186454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:07.186438    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c"
Nov 03 10:37:22.410475 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:37:22.410441    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:37:31.037252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:31.037215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:37:50.454980 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:50.454943    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:37:50.583547 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:50.583509    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:37:52.410623 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:37:52.410559    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:37:56.468007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.467976    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"]
Nov 03 10:37:56.471221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.471203    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:37:56.478837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.478813    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 10:37:56.483218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.483200    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tmk7v\""
Nov 03 10:37:56.487017 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.486995    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"]
Nov 03 10:37:56.494313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.494292    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"]
Nov 03 10:37:56.497383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.497369    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:37:56.506732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.506712    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"]
Nov 03 10:37:56.530721 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.530698    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:37:56.661142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.661105    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3023359d-37a2-4fbf-9241-442ecaf7fd1f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xkl44\" (UID: \"3023359d-37a2-4fbf-9241-442ecaf7fd1f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:37:56.661337 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.661179    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk57x\" (UniqueName: \"kubernetes.io/projected/3023359d-37a2-4fbf-9241-442ecaf7fd1f-kube-api-access-zk57x\") pod \"payload-pre-processing-7b98bb678c-xkl44\" (UID: \"3023359d-37a2-4fbf-9241-442ecaf7fd1f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:37:56.661337 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.661237    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn2hn\" (UniqueName: \"kubernetes.io/projected/99ada003-4afe-4019-8dab-6c9dadf4b888-kube-api-access-qn2hn\") pod \"payload-processing-55fbbbf9dc-2f7kn\" (UID: \"99ada003-4afe-4019-8dab-6c9dadf4b888\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:37:56.661446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.661353    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ada003-4afe-4019-8dab-6c9dadf4b888-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2f7kn\" (UID: \"99ada003-4afe-4019-8dab-6c9dadf4b888\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:37:56.696144 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.696106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:37:56.762456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.762368    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ada003-4afe-4019-8dab-6c9dadf4b888-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2f7kn\" (UID: \"99ada003-4afe-4019-8dab-6c9dadf4b888\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:37:56.762456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.762449    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3023359d-37a2-4fbf-9241-442ecaf7fd1f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xkl44\" (UID: \"3023359d-37a2-4fbf-9241-442ecaf7fd1f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:37:56.762711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.762532    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk57x\" (UniqueName: \"kubernetes.io/projected/3023359d-37a2-4fbf-9241-442ecaf7fd1f-kube-api-access-zk57x\") pod \"payload-pre-processing-7b98bb678c-xkl44\" (UID: \"3023359d-37a2-4fbf-9241-442ecaf7fd1f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:37:56.762711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.762596    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn2hn\" (UniqueName: \"kubernetes.io/projected/99ada003-4afe-4019-8dab-6c9dadf4b888-kube-api-access-qn2hn\") pod \"payload-processing-55fbbbf9dc-2f7kn\" (UID: \"99ada003-4afe-4019-8dab-6c9dadf4b888\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:37:56.763195 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.763164    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3023359d-37a2-4fbf-9241-442ecaf7fd1f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xkl44\" (UID: \"3023359d-37a2-4fbf-9241-442ecaf7fd1f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:37:56.763360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.763212    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ada003-4afe-4019-8dab-6c9dadf4b888-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2f7kn\" (UID: \"99ada003-4afe-4019-8dab-6c9dadf4b888\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:37:56.770566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.770539    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn2hn\" (UniqueName: \"kubernetes.io/projected/99ada003-4afe-4019-8dab-6c9dadf4b888-kube-api-access-qn2hn\") pod \"payload-processing-55fbbbf9dc-2f7kn\" (UID: \"99ada003-4afe-4019-8dab-6c9dadf4b888\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:37:56.770719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.770702    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk57x\" (UniqueName: \"kubernetes.io/projected/3023359d-37a2-4fbf-9241-442ecaf7fd1f-kube-api-access-zk57x\") pod \"payload-pre-processing-7b98bb678c-xkl44\" (UID: \"3023359d-37a2-4fbf-9241-442ecaf7fd1f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:37:56.787494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.787476    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:37:56.811209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:56.811191    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:37:57.064464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:57.064438    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"]
Nov 03 10:37:57.066730 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:37:57.066704    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ada003_4afe_4019_8dab_6c9dadf4b888.slice/crio-58198e2ea052f315e6f63b08f90ad48f40e8e32de2d9b01d05a153dd7592a3fb WatchSource:0}: Error finding container 58198e2ea052f315e6f63b08f90ad48f40e8e32de2d9b01d05a153dd7592a3fb: Status 404 returned error can't find the container with id 58198e2ea052f315e6f63b08f90ad48f40e8e32de2d9b01d05a153dd7592a3fb
Nov 03 10:37:57.091468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:57.091448    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"]
Nov 03 10:37:57.093316 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:37:57.093293    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3023359d_37a2_4fbf_9241_442ecaf7fd1f.slice/crio-245fa7cd9ddfd3f332bf546ca8af057c33100d062c265451df9916e2cccb7f10 WatchSource:0}: Error finding container 245fa7cd9ddfd3f332bf546ca8af057c33100d062c265451df9916e2cccb7f10: Status 404 returned error can't find the container with id 245fa7cd9ddfd3f332bf546ca8af057c33100d062c265451df9916e2cccb7f10
Nov 03 10:37:57.462887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:57.462857    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44" event={"ID":"3023359d-37a2-4fbf-9241-442ecaf7fd1f","Type":"ContainerStarted","Data":"245fa7cd9ddfd3f332bf546ca8af057c33100d062c265451df9916e2cccb7f10"}
Nov 03 10:37:57.463999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:57.463982    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn" event={"ID":"99ada003-4afe-4019-8dab-6c9dadf4b888","Type":"ContainerStarted","Data":"58198e2ea052f315e6f63b08f90ad48f40e8e32de2d9b01d05a153dd7592a3fb"}
Nov 03 10:37:59.870244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:37:59.870161    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:00.483518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:00.483485    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44" event={"ID":"3023359d-37a2-4fbf-9241-442ecaf7fd1f","Type":"ContainerStarted","Data":"028a521f1ee67e792fae0c1b86e7bdae75aca61346464cf47b84bfaedc303995"}
Nov 03 10:38:00.483718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:00.483600    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:38:00.485173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:00.485150    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn" event={"ID":"99ada003-4afe-4019-8dab-6c9dadf4b888","Type":"ContainerStarted","Data":"6abed492bafcc1f7546c8c1570b6168d9015848a42a0fa5512cfbc16fe3d4fda"}
Nov 03 10:38:00.485275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:00.485239    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:38:00.499379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:00.499342    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44" podStartSLOduration=2.091690232 podStartE2EDuration="4.499330727s" totalImagesPullingTime="2.407640495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:00.498051419 +0000 UTC m=+510.703633470" watchObservedRunningTime="2026-11-03 10:38:00.499330727 +0000 UTC m=+510.704912775"
Nov 03 10:38:00.512959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:00.512917    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn" podStartSLOduration=2.039255344 podStartE2EDuration="4.512905071s" totalImagesPullingTime="2.473649727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:00.511716846 +0000 UTC m=+510.717298898" watchObservedRunningTime="2026-11-03 10:38:00.512905071 +0000 UTC m=+510.718487118"
Nov 03 10:38:11.498948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:11.498922    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xkl44"
Nov 03 10:38:11.499504 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:11.498990    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-2f7kn"
Nov 03 10:38:20.807204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:20.807167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:21.291579 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:21.291543    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:22.411341 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:38:22.411302    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:38:22.539924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:22.539891    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:23.868610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:23.868556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:25.922858 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:25.922828    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:26.364802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:26.364769    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:38:28.718755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.718725    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"]
Nov 03 10:38:28.726925 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.726903    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.731873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.731851    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"]
Nov 03 10:38:28.733031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.733010    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 10:38:28.733154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.733040    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 10:38:28.733789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.733772    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4cvtj\""
Nov 03 10:38:28.733789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.733775    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 10:38:28.810695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.810667    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:28.818200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.818177    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.818288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.818212    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.818288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.818228    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042afe48-f39e-4b28-bb65-d7db0e147638-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.818380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.818362    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.818425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.818414    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.818475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.818463    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtdw\" (UniqueName: \"kubernetes.io/projected/042afe48-f39e-4b28-bb65-d7db0e147638-kube-api-access-hrtdw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919259    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919311    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919432    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtdw\" (UniqueName: \"kubernetes.io/projected/042afe48-f39e-4b28-bb65-d7db0e147638-kube-api-access-hrtdw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919491    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919549    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919592    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042afe48-f39e-4b28-bb65-d7db0e147638-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919704    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919808 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919757    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.919868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.919833    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.921979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.921959    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042afe48-f39e-4b28-bb65-d7db0e147638-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.922269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.922250    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042afe48-f39e-4b28-bb65-d7db0e147638-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:28.931600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:28.931559    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtdw\" (UniqueName: \"kubernetes.io/projected/042afe48-f39e-4b28-bb65-d7db0e147638-kube-api-access-hrtdw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2xhzm\" (UID: \"042afe48-f39e-4b28-bb65-d7db0e147638\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:29.041608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:29.041502    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:29.225129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:29.225101    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:38:29.318506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:29.318475    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"]
Nov 03 10:38:29.319755 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:38:29.319727    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042afe48_f39e_4b28_bb65_d7db0e147638.slice/crio-9a6e8b814844245c8f7394cad7fb42eee7217daa7a137dd6e857c82cf209bfde WatchSource:0}: Error finding container 9a6e8b814844245c8f7394cad7fb42eee7217daa7a137dd6e857c82cf209bfde: Status 404 returned error can't find the container with id 9a6e8b814844245c8f7394cad7fb42eee7217daa7a137dd6e857c82cf209bfde
Nov 03 10:38:29.634975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:29.634899    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm" event={"ID":"042afe48-f39e-4b28-bb65-d7db0e147638","Type":"ContainerStarted","Data":"9a6e8b814844245c8f7394cad7fb42eee7217daa7a137dd6e857c82cf209bfde"}
Nov 03 10:38:29.987921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:29.987881    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:31.075431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.075348    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:31.395523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.394902    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"]
Nov 03 10:38:31.401608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.399489    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.410343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.406853    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 10:38:31.411220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.411186    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"]
Nov 03 10:38:31.494641 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.494602    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:31.565842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.565810    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9fm\" (UniqueName: \"kubernetes.io/projected/550263cd-a336-4f27-abad-fafbbc1c61c6-kube-api-access-bd9fm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.566011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.565846    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/550263cd-a336-4f27-abad-fafbbc1c61c6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.566011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.565874    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.566111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.566007    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.566111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.566043    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.566111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.566064    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.648645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.648545    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm" event={"ID":"042afe48-f39e-4b28-bb65-d7db0e147638","Type":"ContainerStarted","Data":"d1c69162b624da3d8086076f0b28dfa219a032783559b42b3429260f32e5869f"}
Nov 03 10:38:31.666384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.666331    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm" podStartSLOduration=2.162541855 podStartE2EDuration="3.666314234s" totalImagesPullingTime="1.503772379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:31.663919127 +0000 UTC m=+541.869501179" watchObservedRunningTime="2026-11-03 10:38:31.666314234 +0000 UTC m=+541.871896290"
Nov 03 10:38:31.666885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.666866    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9fm\" (UniqueName: \"kubernetes.io/projected/550263cd-a336-4f27-abad-fafbbc1c61c6-kube-api-access-bd9fm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.666964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.666895    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/550263cd-a336-4f27-abad-fafbbc1c61c6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.666964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.666913    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.666964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.666948    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.667120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.666993    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.667120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.667021    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.667365 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.667335    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.667446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.667424    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.667520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.667496    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.669494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.669475    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/550263cd-a336-4f27-abad-fafbbc1c61c6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.669606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.669540    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/550263cd-a336-4f27-abad-fafbbc1c61c6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.675135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.675115    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9fm\" (UniqueName: \"kubernetes.io/projected/550263cd-a336-4f27-abad-fafbbc1c61c6-kube-api-access-bd9fm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x9t8q\" (UID: \"550263cd-a336-4f27-abad-fafbbc1c61c6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.732103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.732077    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:31.964249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:31.964220    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:38:32.014457 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.014427    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"]
Nov 03 10:38:32.018027 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:38:32.018001    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550263cd_a336_4f27_abad_fafbbc1c61c6.slice/crio-ad2a917e2ca0ba5c501cecd54c8151c4cb685120322f01bc80d0e6f94a184c22 WatchSource:0}: Error finding container ad2a917e2ca0ba5c501cecd54c8151c4cb685120322f01bc80d0e6f94a184c22: Status 404 returned error can't find the container with id ad2a917e2ca0ba5c501cecd54c8151c4cb685120322f01bc80d0e6f94a184c22
Nov 03 10:38:32.042430 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.042406    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:32.089104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.089063    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:32.650482 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.650398    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lrkj6"]
Nov 03 10:38:32.654802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.654773    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.659912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.659478    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q" event={"ID":"550263cd-a336-4f27-abad-fafbbc1c61c6","Type":"ContainerStarted","Data":"f567948656c950a4dead31431c5825735e4d431279a07810881c2baaef339fb3"}
Nov 03 10:38:32.659912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.659541    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q" event={"ID":"550263cd-a336-4f27-abad-fafbbc1c61c6","Type":"ContainerStarted","Data":"ad2a917e2ca0ba5c501cecd54c8151c4cb685120322f01bc80d0e6f94a184c22"}
Nov 03 10:38:32.661106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.661080    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 10:38:32.664977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.664950    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lrkj6"]
Nov 03 10:38:32.711321 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.711263    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q" podStartSLOduration=1.419764592 podStartE2EDuration="1.711243797s" totalImagesPullingTime="291.479205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:32.709289663 +0000 UTC m=+542.914871717" watchObservedRunningTime="2026-11-03 10:38:32.711243797 +0000 UTC m=+542.916825848"
Nov 03 10:38:32.732958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.732927    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:32.755740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.755679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:32.794519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.794483    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.794519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.794518    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pc9\" (UniqueName: \"kubernetes.io/projected/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-kube-api-access-89pc9\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.794768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.794556    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.794768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.794621    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.794768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.794659    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-home\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.794768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.794739    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.895951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.895922    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.896161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.895971    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-home\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.896161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.896033    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.896161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.896061    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.896161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.896077    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pc9\" (UniqueName: \"kubernetes.io/projected/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-kube-api-access-89pc9\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.896161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.896102    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.896428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.896389    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.896428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.896416    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-home\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.896593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.896550    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.898942 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.898877    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.899463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.899439    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.904562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.904492    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pc9\" (UniqueName: \"kubernetes.io/projected/618e5bc7-d6bb-4234-80c5-a3da85ed8cb2-kube-api-access-89pc9\") pod \"echo-simulator-kserve-64b6f8f96c-lrkj6\" (UID: \"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:32.997373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:32.997331    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:33.285596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:33.285559    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lrkj6"]
Nov 03 10:38:33.668160 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:33.668085    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6" event={"ID":"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2","Type":"ContainerStarted","Data":"668c1b3922b0493d8573229ae4edaa72aae5fce34dffcc1ec359e90ffe10c3fe"}
Nov 03 10:38:33.911174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:33.911129    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:34.612311 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.612279    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"]
Nov 03 10:38:34.621204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.621181    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.623149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.623108    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"]
Nov 03 10:38:34.630039 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.630019    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 10:38:34.755715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.755660    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:34.818794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.818757    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.819100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.818824    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.819100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.818882    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.819100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.818936    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a480b85-cd03-4052-a431-4d092fa8600b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.819597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.819550    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.819689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.819615    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwc5r\" (UniqueName: \"kubernetes.io/projected/0a480b85-cd03-4052-a431-4d092fa8600b-kube-api-access-bwc5r\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.921202 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.921116    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.921202 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.921188    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.921431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.921254    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.921431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.921310    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a480b85-cd03-4052-a431-4d092fa8600b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.921431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.921349    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.921431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.921391    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwc5r\" (UniqueName: \"kubernetes.io/projected/0a480b85-cd03-4052-a431-4d092fa8600b-kube-api-access-bwc5r\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.922298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.921923    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.922298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.922251    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.922489 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.922316    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.928140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.928117    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a480b85-cd03-4052-a431-4d092fa8600b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.932796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.932766    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwc5r\" (UniqueName: \"kubernetes.io/projected/0a480b85-cd03-4052-a431-4d092fa8600b-kube-api-access-bwc5r\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:34.938815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:34.938789    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a480b85-cd03-4052-a431-4d092fa8600b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw\" (UID: \"0a480b85-cd03-4052-a431-4d092fa8600b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:35.237687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:35.237654    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:35.516669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:35.516639    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"]
Nov 03 10:38:35.519186 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:38:35.519162    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a480b85_cd03_4052_a431_4d092fa8600b.slice/crio-e2d28692c4beeeffb8ee8fcc678da4b6c1138151137754c87552d14f5da9f05b WatchSource:0}: Error finding container e2d28692c4beeeffb8ee8fcc678da4b6c1138151137754c87552d14f5da9f05b: Status 404 returned error can't find the container with id e2d28692c4beeeffb8ee8fcc678da4b6c1138151137754c87552d14f5da9f05b
Nov 03 10:38:35.521276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:35.521255    5817 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:38:35.679888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:35.679857    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" event={"ID":"0a480b85-cd03-4052-a431-4d092fa8600b","Type":"ContainerStarted","Data":"e2d28692c4beeeffb8ee8fcc678da4b6c1138151137754c87552d14f5da9f05b"}
Nov 03 10:38:35.681700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:35.681675    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6" event={"ID":"618e5bc7-d6bb-4234-80c5-a3da85ed8cb2","Type":"ContainerStarted","Data":"553e008d1438fea669e7235f04173bebff4dcba51090c28ca64c11628b9d8c5d"}
Nov 03 10:38:35.699108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:35.699054    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6" podStartSLOduration=1.951037377 podStartE2EDuration="3.699039331s" totalImagesPullingTime="1.748001954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:35.697263222 +0000 UTC m=+545.902845271" watchObservedRunningTime="2026-11-03 10:38:35.699039331 +0000 UTC m=+545.904621382"
Nov 03 10:38:35.998215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:35.998183    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:36.351812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:36.351776    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:38:36.688500 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:36.688470    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" event={"ID":"0a480b85-cd03-4052-a431-4d092fa8600b","Type":"ContainerStarted","Data":"1a7cf2760e71592e5fb2fc799762e7fdd89c61cd14e100eef0fa342bf470d457"}
Nov 03 10:38:36.705354 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:36.705307    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podStartSLOduration=2.406408258 podStartE2EDuration="2.705292592s" totalImagesPullingTime="298.884334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:36.705167713 +0000 UTC m=+546.910749764" watchObservedRunningTime="2026-11-03 10:38:36.705292592 +0000 UTC m=+546.910874642"
Nov 03 10:38:37.157715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:37.157625    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:37.237899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:37.237868    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:39.042202 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:39.042169    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:39.055376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:39.055350    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:39.720896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:39.720870    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm"
Nov 03 10:38:41.732764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:41.732728    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:41.744964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:41.744942    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:42.736088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:42.736062    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q"
Nov 03 10:38:42.997956 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:42.997882    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:43.010280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:43.010263    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:43.746190 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:43.746163    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6"
Nov 03 10:38:45.238527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:45.238495    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:45.251072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:45.251050    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:45.753141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:45.753110    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:38:50.300140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:50.300106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:50.461364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:50.461318    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:52.411281 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:38:52.411244    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:38:54.207256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.207220    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"]
Nov 03 10:38:54.211083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.211063    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.225007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.224986    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"]
Nov 03 10:38:54.288765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.288733    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fec13f-2b85-413d-88a6-de064830453b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.288896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.288795    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.288938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.288898    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fec13f-2b85-413d-88a6-de064830453b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.288938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.288927    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.289020 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.288963    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.289070 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.289051    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcbx\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-kube-api-access-mdcbx\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.289125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.289084    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.289200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.289181    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.289260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.289221    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390265    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fec13f-2b85-413d-88a6-de064830453b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390303    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390331    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390353    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcbx\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-kube-api-access-mdcbx\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390371    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390440    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390788    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390800    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390824    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.390903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390888    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fec13f-2b85-413d-88a6-de064830453b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.391102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390901    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.391102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.390933    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.391366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.391234    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.391427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.391368    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fec13f-2b85-413d-88a6-de064830453b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.393053 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.393022    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.393171 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.393116    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fec13f-2b85-413d-88a6-de064830453b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.398518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.398491    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcbx\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-kube-api-access-mdcbx\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.398640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.398560    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-b9tw2\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.530176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.530091    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:54.816708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.816686    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"]
Nov 03 10:38:54.818837 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:38:54.818809    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fec13f_2b85_413d_88a6_de064830453b.slice/crio-a232c93d9953e57931b0d533755fac2cb76d3d7f0aac1dea60f58277affbd015 WatchSource:0}: Error finding container a232c93d9953e57931b0d533755fac2cb76d3d7f0aac1dea60f58277affbd015: Status 404 returned error can't find the container with id a232c93d9953e57931b0d533755fac2cb76d3d7f0aac1dea60f58277affbd015
Nov 03 10:38:54.821353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.821322    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:38:54.821448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.821385    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:38:54.821448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:54.821414    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:38:55.813119 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:55.813076    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" event={"ID":"51fec13f-2b85-413d-88a6-de064830453b","Type":"ContainerStarted","Data":"dc0693e46931adce7de223b178ff6df4b13c4f3a51c614867fa6e7fcf0c78646"}
Nov 03 10:38:55.813735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:55.813709    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" event={"ID":"51fec13f-2b85-413d-88a6-de064830453b","Type":"ContainerStarted","Data":"a232c93d9953e57931b0d533755fac2cb76d3d7f0aac1dea60f58277affbd015"}
Nov 03 10:38:55.836889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:55.836838    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podStartSLOduration=1.836822341 podStartE2EDuration="1.836822341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:55.835270054 +0000 UTC m=+566.040852107" watchObservedRunningTime="2026-11-03 10:38:55.836822341 +0000 UTC m=+566.042404393"
Nov 03 10:38:56.531158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:56.531127    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:38:56.532538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:56.532510    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:38:56.532687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:56.532567    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 10:38:57.326445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:57.326396    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:57.531471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:57.531436    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:38:57.531670 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:57.531520    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 10:38:58.042073 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:58.042034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:38:58.530638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:58.530605    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:38:58.531028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:58.530684    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 10:38:59.531203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:59.531168    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:38:59.531706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:38:59.531240    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 10:39:00.530920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:00.530882    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:39:00.531102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:00.530943    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 10:39:01.530622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:01.530566    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:39:01.531081 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:01.530646    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 10:39:02.065085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:02.065046    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:39:02.200415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:02.200379    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:39:02.998469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:02.998430    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:39:02.998947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:02.998798    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:39:02.999717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:02.999694    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 10:39:04.889324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:04.889291    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:39:09.214232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.214200    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"]
Nov 03 10:39:09.221036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.221012    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.232618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.232593    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"]
Nov 03 10:39:09.331604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.331558    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjz5\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-kube-api-access-mpjz5\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.331760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.331645    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.331760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.331691    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acd137a-8e3c-4b9e-a863-73764a7f6990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.331760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.331708    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.331928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.331833    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.331928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.331892    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.332018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.331929    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.332018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.331992    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.332018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.332015    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433318    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433362    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433385    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433412    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433441    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433620    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjz5\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-kube-api-access-mpjz5\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433662    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433723    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acd137a-8e3c-4b9e-a863-73764a7f6990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433749    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433770    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433781    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.433991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.433844    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.434106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.434084    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.434344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.434326    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acd137a-8e3c-4b9e-a863-73764a7f6990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.435922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.435896    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.436201 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.436176    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.440867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.440850    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.441303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.441280    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjz5\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-kube-api-access-mpjz5\") pod \"maas-default-gateway-openshift-default-6cc458d887-7h7sb\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.537536 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.537478    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:09.813335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.813244    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"]
Nov 03 10:39:09.815323 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:39:09.815298    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acd137a_8e3c_4b9e_a863_73764a7f6990.slice/crio-f5db00308aacc1532c4493d1a0bdf7597ba7da0287c20b29b2b04f88c72ce74a WatchSource:0}: Error finding container f5db00308aacc1532c4493d1a0bdf7597ba7da0287c20b29b2b04f88c72ce74a: Status 404 returned error can't find the container with id f5db00308aacc1532c4493d1a0bdf7597ba7da0287c20b29b2b04f88c72ce74a
Nov 03 10:39:09.817590 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.817541    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:09.817688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.817620    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:09.817688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.817648    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:09.893179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.893147    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" event={"ID":"3acd137a-8e3c-4b9e-a863-73764a7f6990","Type":"ContainerStarted","Data":"4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e"}
Nov 03 10:39:09.893292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.893203    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" event={"ID":"3acd137a-8e3c-4b9e-a863-73764a7f6990","Type":"ContainerStarted","Data":"f5db00308aacc1532c4493d1a0bdf7597ba7da0287c20b29b2b04f88c72ce74a"}
Nov 03 10:39:09.913058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:09.913001    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podStartSLOduration=0.912983178 podStartE2EDuration="912.983178ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:39:09.909750103 +0000 UTC m=+580.115332154" watchObservedRunningTime="2026-11-03 10:39:09.912983178 +0000 UTC m=+580.118565230"
Nov 03 10:39:10.538298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:10.538260    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:10.539984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:10.539952    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7h7sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 10:39:10.540088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:10.540022    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 10:39:11.538022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:11.537988    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7h7sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 10:39:11.538229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:11.538053    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 10:39:12.538047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:12.538015    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7h7sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 10:39:12.538441 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:12.538064    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 10:39:13.538139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:13.538104    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7h7sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 10:39:13.538522 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:13.538154    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 10:39:14.538458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:14.538420    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7h7sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 10:39:14.538878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:14.538472    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 10:39:15.366008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:15.365967    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:39:15.506595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:15.505558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:39:15.538758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:15.538729    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7h7sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 10:39:15.539111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:15.538780    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 10:39:16.538548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:16.538510    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7h7sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 10:39:16.538760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:16.538584    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 10:39:17.969018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:17.968985    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:17.969486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:17.969395    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:17.970347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:17.970324    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:39:22.409996 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:39:22.409958    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:39:31.150776 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:31.150739    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:39:41.130274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:41.130240    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:39:46.169864 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:39:46.169819    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:39:52.410784 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:39:52.410749    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:40:00.803941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:00.803893    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:00.944383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:00.944338    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:01.087191 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.087103    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:01.370494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.370413    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:01.412784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.412753    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"]
Nov 03 10:40:01.416669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.416637    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.420166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.420145    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3a6822cf-openshift-default-dockercfg-j5l9c\""
Nov 03 10:40:01.428143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.427869    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"]
Nov 03 10:40:01.502874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.502844    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fec804-7104-45c8-8c9b-9a901c1505eb-istiod-ca-cert\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.503026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.502883    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-podinfo\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.503026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.502972    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jz79\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-kube-api-access-5jz79\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.503026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.503020    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-data\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.503184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.503038    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-certs\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.503184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.503054    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-socket\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.503184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.503173    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-credential-socket\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.503322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.503215    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-envoy\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.503322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.503282    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-token\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.566484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.566426    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:01.604377 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604349    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-token\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.604532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604400    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fec804-7104-45c8-8c9b-9a901c1505eb-istiod-ca-cert\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.604532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604443    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-podinfo\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.604532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604488    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jz79\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-kube-api-access-5jz79\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.604730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604558    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-data\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.604785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604715    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-certs\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.604785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604761    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-socket\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.604885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604808    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-credential-socket\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.604885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.604857    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-envoy\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.605054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.605034    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-data\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.605265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.605222    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-socket\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.605388 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.605267    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fec804-7104-45c8-8c9b-9a901c1505eb-istiod-ca-cert\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.605449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.605410    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-certs\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.605506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.605442    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-credential-socket\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.607259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.607240    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-podinfo\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.607357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.607340    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-envoy\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.613686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.613661    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jz79\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-kube-api-access-5jz79\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.613686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.613673    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-token\") pod \"e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") " pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.704206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.704177    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:01.730737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.730712    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:01.833508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.833364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:01.964603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:01.964481    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:02.049161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:02.049139    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"]
Nov 03 10:40:02.051053 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:02.051028    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fec804_7104_45c8_8c9b_9a901c1505eb.slice/crio-e50184f0f3d7b226ace91a3bb1444b690e60dc6cd4df1ac3fc29d659f7af8695 WatchSource:0}: Error finding container e50184f0f3d7b226ace91a3bb1444b690e60dc6cd4df1ac3fc29d659f7af8695: Status 404 returned error can't find the container with id e50184f0f3d7b226ace91a3bb1444b690e60dc6cd4df1ac3fc29d659f7af8695
Nov 03 10:40:02.053560 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:02.053526    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:02.053673 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:02.053618    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:02.053744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:02.053681    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:02.180157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:02.180115    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" event={"ID":"a0fec804-7104-45c8-8c9b-9a901c1505eb","Type":"ContainerStarted","Data":"3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242"}
Nov 03 10:40:02.180338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:02.180184    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" event={"ID":"a0fec804-7104-45c8-8c9b-9a901c1505eb","Type":"ContainerStarted","Data":"e50184f0f3d7b226ace91a3bb1444b690e60dc6cd4df1ac3fc29d659f7af8695"}
Nov 03 10:40:02.201198 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:02.201144    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podStartSLOduration=1.201126146 podStartE2EDuration="1.201126146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:02.200461893 +0000 UTC m=+632.406043946" watchObservedRunningTime="2026-11-03 10:40:02.201126146 +0000 UTC m=+632.406708199"
Nov 03 10:40:02.731677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:02.731642    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:03.738055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:03.738027    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:03.764738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:03.764694    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:03.909112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:03.909067    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:04.189941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.189909    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:04.191138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.191116    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:04.403894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.403840    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:04.532892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.532798    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:04.640613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.640562    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"]
Nov 03 10:40:04.644805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.644784    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.654791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.654766    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3baa762a-serving-cert\""
Nov 03 10:40:04.659623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.655509    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l8bvs\""
Nov 03 10:40:04.659623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.655538    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 10:40:04.659623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.655551    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 10:40:04.659623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.655551    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 10:40:04.659623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.658618    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"]
Nov 03 10:40:04.739987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.739959    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwk7m\" (UniqueName: \"kubernetes.io/projected/e4dd2013-aa16-4a4c-920c-7a8795a549ae-kube-api-access-rwk7m\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.740415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.740007    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-maas-api-tls\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.740415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.740286    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd2013-aa16-4a4c-920c-7a8795a549ae-openshift-service-ca\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.740415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.740379    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-metrics-certs\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.752683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.752649    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:04.841935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.841858    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwk7m\" (UniqueName: \"kubernetes.io/projected/e4dd2013-aa16-4a4c-920c-7a8795a549ae-kube-api-access-rwk7m\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.841935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.841890    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-maas-api-tls\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.842160 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.841969    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd2013-aa16-4a4c-920c-7a8795a549ae-openshift-service-ca\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.842160 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.842040    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-metrics-certs\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.842831 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.842717    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd2013-aa16-4a4c-920c-7a8795a549ae-openshift-service-ca\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.844688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.844661    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-metrics-certs\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.844794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.844667    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-maas-api-tls\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.850180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.850156    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"]
Nov 03 10:40:04.851090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.851066    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwk7m\" (UniqueName: \"kubernetes.io/projected/e4dd2013-aa16-4a4c-920c-7a8795a549ae-kube-api-access-rwk7m\") pod \"maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:04.853853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.853836    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:04.862332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.862299    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3baa762a\""
Nov 03 10:40:04.862465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.862450    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"]
Nov 03 10:40:04.898364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.898333    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:04.919279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.919252    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"]
Nov 03 10:40:04.922643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.922626    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:04.929990 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.929969    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3baa762a-dockercfg-qcr82\""
Nov 03 10:40:04.934233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.934205    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"]
Nov 03 10:40:04.942981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.942960    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjdj\" (UniqueName: \"kubernetes.io/projected/3e5226b0-0969-41f1-99b0-308e930c1211-kube-api-access-qfjdj\") pod \"payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g\" (UID: \"3e5226b0-0969-41f1-99b0-308e930c1211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:04.943163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.943148    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5226b0-0969-41f1-99b0-308e930c1211-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g\" (UID: \"3e5226b0-0969-41f1-99b0-308e930c1211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:04.959899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:04.959881    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:05.044013 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.043970    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjdj\" (UniqueName: \"kubernetes.io/projected/3e5226b0-0969-41f1-99b0-308e930c1211-kube-api-access-qfjdj\") pod \"payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g\" (UID: \"3e5226b0-0969-41f1-99b0-308e930c1211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:05.044192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.044066    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhdf\" (UniqueName: \"kubernetes.io/projected/3a32d224-89de-4f3c-9230-cdbd76db9d60-kube-api-access-fzhdf\") pod \"payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j\" (UID: \"3a32d224-89de-4f3c-9230-cdbd76db9d60\") " pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:05.044192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.044099    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a32d224-89de-4f3c-9230-cdbd76db9d60-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j\" (UID: \"3a32d224-89de-4f3c-9230-cdbd76db9d60\") " pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:05.044308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.044226    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5226b0-0969-41f1-99b0-308e930c1211-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g\" (UID: \"3e5226b0-0969-41f1-99b0-308e930c1211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:05.046048 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.045024    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5226b0-0969-41f1-99b0-308e930c1211-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g\" (UID: \"3e5226b0-0969-41f1-99b0-308e930c1211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:05.057674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.057463    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjdj\" (UniqueName: \"kubernetes.io/projected/3e5226b0-0969-41f1-99b0-308e930c1211-kube-api-access-qfjdj\") pod \"payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g\" (UID: \"3e5226b0-0969-41f1-99b0-308e930c1211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:05.145002 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.144930    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a32d224-89de-4f3c-9230-cdbd76db9d60-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j\" (UID: \"3a32d224-89de-4f3c-9230-cdbd76db9d60\") " pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:05.145125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.145087    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhdf\" (UniqueName: \"kubernetes.io/projected/3a32d224-89de-4f3c-9230-cdbd76db9d60-kube-api-access-fzhdf\") pod \"payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j\" (UID: \"3a32d224-89de-4f3c-9230-cdbd76db9d60\") " pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:05.145740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.145722    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a32d224-89de-4f3c-9230-cdbd76db9d60-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j\" (UID: \"3a32d224-89de-4f3c-9230-cdbd76db9d60\") " pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:05.160075 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.160051    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhdf\" (UniqueName: \"kubernetes.io/projected/3a32d224-89de-4f3c-9230-cdbd76db9d60-kube-api-access-fzhdf\") pod \"payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j\" (UID: \"3a32d224-89de-4f3c-9230-cdbd76db9d60\") " pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:05.176927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.176909    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:05.238217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.238183    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:05.287387 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:05.287357    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dd2013_aa16_4a4c_920c_7a8795a549ae.slice/crio-cffab71850d11d50aac955f0f7f72355f1b06ac1fcf0464222886bd70913c01d WatchSource:0}: Error finding container cffab71850d11d50aac955f0f7f72355f1b06ac1fcf0464222886bd70913c01d: Status 404 returned error can't find the container with id cffab71850d11d50aac955f0f7f72355f1b06ac1fcf0464222886bd70913c01d
Nov 03 10:40:05.301725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.296396    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"]
Nov 03 10:40:05.453972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.453945    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"]
Nov 03 10:40:05.456346 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:05.456309    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5226b0_0969_41f1_99b0_308e930c1211.slice/crio-ef2377d4ece2d20486ec202bf4a6aab9b77ea65a90a563bfc213acd7d978d4f1 WatchSource:0}: Error finding container ef2377d4ece2d20486ec202bf4a6aab9b77ea65a90a563bfc213acd7d978d4f1: Status 404 returned error can't find the container with id ef2377d4ece2d20486ec202bf4a6aab9b77ea65a90a563bfc213acd7d978d4f1
Nov 03 10:40:05.537141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.537116    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"]
Nov 03 10:40:05.538910 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:05.538883    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a32d224_89de_4f3c_9230_cdbd76db9d60.slice/crio-0ca9677bb58ecace344241d722a3cb2e9d1fe893b3a1d13d0c1c25794811111e WatchSource:0}: Error finding container 0ca9677bb58ecace344241d722a3cb2e9d1fe893b3a1d13d0c1c25794811111e: Status 404 returned error can't find the container with id 0ca9677bb58ecace344241d722a3cb2e9d1fe893b3a1d13d0c1c25794811111e
Nov 03 10:40:05.593778 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.593749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:05.607364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.607340    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"]
Nov 03 10:40:05.612795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.612772    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:05.617947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.616955    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"]
Nov 03 10:40:05.651631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.651607    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eaf866-9a83-4df3-950e-97299af275b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb\" (UID: \"46eaf866-9a83-4df3-950e-97299af275b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:05.651750 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.651679    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khxcq\" (UniqueName: \"kubernetes.io/projected/46eaf866-9a83-4df3-950e-97299af275b5-kube-api-access-khxcq\") pod \"payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb\" (UID: \"46eaf866-9a83-4df3-950e-97299af275b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:05.678183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.678164    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3a6822cf\""
Nov 03 10:40:05.687950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.687928    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3a6822cf-dockercfg-858hf\""
Nov 03 10:40:05.746707 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.746629    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:05.753898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.753874    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eaf866-9a83-4df3-950e-97299af275b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb\" (UID: \"46eaf866-9a83-4df3-950e-97299af275b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:05.754011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.753941    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khxcq\" (UniqueName: \"kubernetes.io/projected/46eaf866-9a83-4df3-950e-97299af275b5-kube-api-access-khxcq\") pod \"payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb\" (UID: \"46eaf866-9a83-4df3-950e-97299af275b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:05.754589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.754554    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eaf866-9a83-4df3-950e-97299af275b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb\" (UID: \"46eaf866-9a83-4df3-950e-97299af275b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:05.771744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.771712    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khxcq\" (UniqueName: \"kubernetes.io/projected/46eaf866-9a83-4df3-950e-97299af275b5-kube-api-access-khxcq\") pod \"payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb\" (UID: \"46eaf866-9a83-4df3-950e-97299af275b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:05.903812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.903783    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:05.933724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:05.933693    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:06.214413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.213325    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"]
Nov 03 10:40:06.214413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.213393    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9" event={"ID":"e4dd2013-aa16-4a4c-920c-7a8795a549ae","Type":"ContainerStarted","Data":"cffab71850d11d50aac955f0f7f72355f1b06ac1fcf0464222886bd70913c01d"}
Nov 03 10:40:06.215654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.215490    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g" event={"ID":"3e5226b0-0969-41f1-99b0-308e930c1211","Type":"ContainerStarted","Data":"8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5"}
Nov 03 10:40:06.215654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.215545    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g" event={"ID":"3e5226b0-0969-41f1-99b0-308e930c1211","Type":"ContainerStarted","Data":"ef2377d4ece2d20486ec202bf4a6aab9b77ea65a90a563bfc213acd7d978d4f1"}
Nov 03 10:40:06.215654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.215606    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:06.217127 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:06.217101    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46eaf866_9a83_4df3_950e_97299af275b5.slice/crio-8ce1029bf7b7b2d01a1f2871c4ca5346df255730b79c40847c746e1c7241d141 WatchSource:0}: Error finding container 8ce1029bf7b7b2d01a1f2871c4ca5346df255730b79c40847c746e1c7241d141: Status 404 returned error can't find the container with id 8ce1029bf7b7b2d01a1f2871c4ca5346df255730b79c40847c746e1c7241d141
Nov 03 10:40:06.220310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.219381    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j" event={"ID":"3a32d224-89de-4f3c-9230-cdbd76db9d60","Type":"ContainerStarted","Data":"c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118"}
Nov 03 10:40:06.220310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.219416    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j" event={"ID":"3a32d224-89de-4f3c-9230-cdbd76db9d60","Type":"ContainerStarted","Data":"0ca9677bb58ecace344241d722a3cb2e9d1fe893b3a1d13d0c1c25794811111e"}
Nov 03 10:40:06.220310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.219443    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:06.235619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.235171    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g" podStartSLOduration=2.235156263 podStartE2EDuration="2.235156263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:06.233937844 +0000 UTC m=+636.439519901" watchObservedRunningTime="2026-11-03 10:40:06.235156263 +0000 UTC m=+636.440738312"
Nov 03 10:40:06.253511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.253471    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j" podStartSLOduration=2.25345995 podStartE2EDuration="2.25345995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:06.252648058 +0000 UTC m=+636.458230110" watchObservedRunningTime="2026-11-03 10:40:06.25345995 +0000 UTC m=+636.459042001"
Nov 03 10:40:06.479635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.479545    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:06.988525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:06.988494    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:40:07.071265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:07.071234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:07.203796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:07.203691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:07.230737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:07.230705    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb" event={"ID":"46eaf866-9a83-4df3-950e-97299af275b5","Type":"ContainerStarted","Data":"1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136"}
Nov 03 10:40:07.230893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:07.230758    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb" event={"ID":"46eaf866-9a83-4df3-950e-97299af275b5","Type":"ContainerStarted","Data":"8ce1029bf7b7b2d01a1f2871c4ca5346df255730b79c40847c746e1c7241d141"}
Nov 03 10:40:07.231305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:07.231275    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:07.246800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:07.246705    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb" podStartSLOduration=2.246693559 podStartE2EDuration="2.246693559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:07.246344166 +0000 UTC m=+637.451926217" watchObservedRunningTime="2026-11-03 10:40:07.246693559 +0000 UTC m=+637.452275609"
Nov 03 10:40:07.818480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:07.818440    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:07.963223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:07.963151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:08.133567 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.133491    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:08.237619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.237540    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9" event={"ID":"e4dd2013-aa16-4a4c-920c-7a8795a549ae","Type":"ContainerStarted","Data":"0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4"}
Nov 03 10:40:08.237822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.237804    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:08.258203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.258160    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9" podStartSLOduration=1.405183584 podStartE2EDuration="4.25814601s" totalImagesPullingTime="2.852962426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:08.257249189 +0000 UTC m=+638.462831242" watchObservedRunningTime="2026-11-03 10:40:08.25814601 +0000 UTC m=+638.463728061"
Nov 03 10:40:08.450026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.449999    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"]
Nov 03 10:40:08.464251 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.464220    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.464837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.464809    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"]
Nov 03 10:40:08.468215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.468190    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fc81ef02-openshift-default-dockercfg-cwb2z\""
Nov 03 10:40:08.534750 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.534707    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:08.602764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.602699    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-certs\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.602937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.602846    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-token\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.602937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.602883    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-data\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.603086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.602969    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-socket\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.603086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.603041    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istiod-ca-cert\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.603184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.603114    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-envoy\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.603229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.603197    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-podinfo\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.603262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.603236    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-credential-socket\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.603262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.603255    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrq98\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-kube-api-access-rrq98\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.671663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.671637    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:08.703918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.703821    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-credential-socket\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.703918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.703860    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq98\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-kube-api-access-rrq98\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.703918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.703887    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-certs\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.703918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.703908    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-token\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.703930    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-data\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.703951    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-socket\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.703994    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istiod-ca-cert\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.704086    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-envoy\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.704121    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-podinfo\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.704441    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-credential-socket\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.704462    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-data\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.704547    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-socket\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.704599    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-certs\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.704871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.704836    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istiod-ca-cert\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.706980 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.706955    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-podinfo\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.707183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.707164    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-envoy\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.712012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.711983    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrq98\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-kube-api-access-rrq98\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.712358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.712331    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-token\") pod \"e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") " pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.780816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.780790    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:08.804335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:08.804309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:09.081843 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.081810    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"]
Nov 03 10:40:09.082954 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:09.082925    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20561cb0_7e1d_4e2a_a8de_ba7aa6202ee8.slice/crio-326b64dd91fa5b58b1f92566cc3870c877af0444e0041745afee7dafacb26ea0 WatchSource:0}: Error finding container 326b64dd91fa5b58b1f92566cc3870c877af0444e0041745afee7dafacb26ea0: Status 404 returned error can't find the container with id 326b64dd91fa5b58b1f92566cc3870c877af0444e0041745afee7dafacb26ea0
Nov 03 10:40:09.085321 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.085290    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:09.085402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.085352    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:09.085402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.085382    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:09.246487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.246450    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" event={"ID":"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8","Type":"ContainerStarted","Data":"558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826"}
Nov 03 10:40:09.246996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.246512    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" event={"ID":"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8","Type":"ContainerStarted","Data":"326b64dd91fa5b58b1f92566cc3870c877af0444e0041745afee7dafacb26ea0"}
Nov 03 10:40:09.268035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.267973    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podStartSLOduration=1.267952453 podStartE2EDuration="1.267952453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:09.265428197 +0000 UTC m=+639.471010249" watchObservedRunningTime="2026-11-03 10:40:09.267952453 +0000 UTC m=+639.473534509"
Nov 03 10:40:09.359541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.359442    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:09.781159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:09.781126    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:10.350182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:10.350145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:10.790599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:10.789104    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:10.840394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:10.840354    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:11.012600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:11.012526    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:11.152948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:11.152857    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:11.256866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:11.256839    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:11.257811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:11.257794    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:11.453346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:11.453312    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:11.605911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:11.605880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:11.774989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:11.774896    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:12.044384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.044311    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"]
Nov 03 10:40:12.048501 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.048479    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:12.057447 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.057420    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fc81ef02\""
Nov 03 10:40:12.057932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.057910    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"]
Nov 03 10:40:12.067047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.067010    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:12.067449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.067423    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"]
Nov 03 10:40:12.080153 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.080111    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"]
Nov 03 10:40:12.080291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.080196    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:12.084145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.084120    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fc81ef02-dockercfg-592df\""
Nov 03 10:40:12.144210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.144185    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b81ee0-c86f-4ebc-94e4-686d3357642f-plugins-config-volume\") pod \"payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8\" (UID: \"80b81ee0-c86f-4ebc-94e4-686d3357642f\") " pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:12.144367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.144268    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7kp\" (UniqueName: \"kubernetes.io/projected/80b81ee0-c86f-4ebc-94e4-686d3357642f-kube-api-access-9p7kp\") pod \"payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8\" (UID: \"80b81ee0-c86f-4ebc-94e4-686d3357642f\") " pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:12.210610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.210552    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:12.245235 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.245196    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b81ee0-c86f-4ebc-94e4-686d3357642f-plugins-config-volume\") pod \"payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8\" (UID: \"80b81ee0-c86f-4ebc-94e4-686d3357642f\") " pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:12.245425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.245249    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44d293f-a431-4bd0-a278-29a79593a981-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn\" (UID: \"b44d293f-a431-4bd0-a278-29a79593a981\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:12.245425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.245280    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5tz\" (UniqueName: \"kubernetes.io/projected/b44d293f-a431-4bd0-a278-29a79593a981-kube-api-access-7s5tz\") pod \"payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn\" (UID: \"b44d293f-a431-4bd0-a278-29a79593a981\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:12.245425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.245410    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7kp\" (UniqueName: \"kubernetes.io/projected/80b81ee0-c86f-4ebc-94e4-686d3357642f-kube-api-access-9p7kp\") pod \"payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8\" (UID: \"80b81ee0-c86f-4ebc-94e4-686d3357642f\") " pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:12.246109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.246088    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b81ee0-c86f-4ebc-94e4-686d3357642f-plugins-config-volume\") pod \"payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8\" (UID: \"80b81ee0-c86f-4ebc-94e4-686d3357642f\") " pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:12.254124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.254104    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7kp\" (UniqueName: \"kubernetes.io/projected/80b81ee0-c86f-4ebc-94e4-686d3357642f-kube-api-access-9p7kp\") pod \"payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8\" (UID: \"80b81ee0-c86f-4ebc-94e4-686d3357642f\") " pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:12.346539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.346441    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44d293f-a431-4bd0-a278-29a79593a981-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn\" (UID: \"b44d293f-a431-4bd0-a278-29a79593a981\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:12.346539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.346485    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5tz\" (UniqueName: \"kubernetes.io/projected/b44d293f-a431-4bd0-a278-29a79593a981-kube-api-access-7s5tz\") pod \"payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn\" (UID: \"b44d293f-a431-4bd0-a278-29a79593a981\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:12.347306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.347283    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44d293f-a431-4bd0-a278-29a79593a981-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn\" (UID: \"b44d293f-a431-4bd0-a278-29a79593a981\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:12.354994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.354967    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5tz\" (UniqueName: \"kubernetes.io/projected/b44d293f-a431-4bd0-a278-29a79593a981-kube-api-access-7s5tz\") pod \"payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn\" (UID: \"b44d293f-a431-4bd0-a278-29a79593a981\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:12.356403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.356378    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:12.370319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.370300    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:12.423211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.423177    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:12.676997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.676968    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"]
Nov 03 10:40:12.678216 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:12.678180    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b81ee0_c86f_4ebc_94e4_686d3357642f.slice/crio-4a808c70be9ca82cf31d8d64fc422bf9676884f1b8c0feb2a71ed6e5e4bad7af WatchSource:0}: Error finding container 4a808c70be9ca82cf31d8d64fc422bf9676884f1b8c0feb2a71ed6e5e4bad7af: Status 404 returned error can't find the container with id 4a808c70be9ca82cf31d8d64fc422bf9676884f1b8c0feb2a71ed6e5e4bad7af
Nov 03 10:40:12.741414 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:12.741388    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44d293f_a431_4bd0_a278_29a79593a981.slice/crio-64c130c37f90bba5bd78bd8e9f969dcb34e9f60743d7d36aa54b5cf7b8f31f45 WatchSource:0}: Error finding container 64c130c37f90bba5bd78bd8e9f969dcb34e9f60743d7d36aa54b5cf7b8f31f45: Status 404 returned error can't find the container with id 64c130c37f90bba5bd78bd8e9f969dcb34e9f60743d7d36aa54b5cf7b8f31f45
Nov 03 10:40:12.743291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.743270    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"]
Nov 03 10:40:12.747889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.747798    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:12.992543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:12.992236    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:13.138949 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.138908    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:13.269584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.269487    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8" event={"ID":"80b81ee0-c86f-4ebc-94e4-686d3357642f","Type":"ContainerStarted","Data":"f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8"}
Nov 03 10:40:13.269584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.269545    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8" event={"ID":"80b81ee0-c86f-4ebc-94e4-686d3357642f","Type":"ContainerStarted","Data":"4a808c70be9ca82cf31d8d64fc422bf9676884f1b8c0feb2a71ed6e5e4bad7af"}
Nov 03 10:40:13.269767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.269705    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:13.271218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.271189    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn" event={"ID":"b44d293f-a431-4bd0-a278-29a79593a981","Type":"ContainerStarted","Data":"49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5"}
Nov 03 10:40:13.271313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.271237    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn" event={"ID":"b44d293f-a431-4bd0-a278-29a79593a981","Type":"ContainerStarted","Data":"64c130c37f90bba5bd78bd8e9f969dcb34e9f60743d7d36aa54b5cf7b8f31f45"}
Nov 03 10:40:13.322898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.322865    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:13.324118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.324037    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8" podStartSLOduration=1.3240197949999999 podStartE2EDuration="1.324019795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:13.312833622 +0000 UTC m=+643.518415677" watchObservedRunningTime="2026-11-03 10:40:13.324019795 +0000 UTC m=+643.529601847"
Nov 03 10:40:13.332429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.332238    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn" podStartSLOduration=1.33222081 podStartE2EDuration="1.33222081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:13.330643112 +0000 UTC m=+643.536225162" watchObservedRunningTime="2026-11-03 10:40:13.33222081 +0000 UTC m=+643.537802861"
Nov 03 10:40:13.406597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.406505    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:40:13.477611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:13.477551    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:14.065369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:14.065330    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:14.254005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:14.253976    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:14.278388 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:14.278358    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:14.575619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:14.575582    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:14.717909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:14.717873    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:15.031317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:15.031278    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:15.524947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:15.524909    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:15.668515 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:15.668482    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:15.953092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:15.953054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:16.197980 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.197944    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"]
Nov 03 10:40:16.202725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.202703    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:16.208671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.208599    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"]
Nov 03 10:40:16.211532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.211453    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-19708dae\""
Nov 03 10:40:16.212661 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.212640    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"]
Nov 03 10:40:16.212923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.212746    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:16.215784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.215251    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:16.225043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.224438    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"]
Nov 03 10:40:16.228316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.228293    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-19708dae-dockercfg-8dntl\""
Nov 03 10:40:16.287219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.287194    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxq6w\" (UniqueName: \"kubernetes.io/projected/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-kube-api-access-hxq6w\") pod \"payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr\" (UID: \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\") " pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:16.287362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.287250    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9235dbab-c8e3-4786-8789-46f194cd9937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6\" (UID: \"9235dbab-c8e3-4786-8789-46f194cd9937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:16.287433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.287351    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxfr\" (UniqueName: \"kubernetes.io/projected/9235dbab-c8e3-4786-8789-46f194cd9937-kube-api-access-qbxfr\") pod \"payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6\" (UID: \"9235dbab-c8e3-4786-8789-46f194cd9937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:16.287433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.287416    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-plugins-config-volume\") pod \"payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr\" (UID: \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\") " pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:16.353248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.353213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:16.388779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.388749    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxq6w\" (UniqueName: \"kubernetes.io/projected/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-kube-api-access-hxq6w\") pod \"payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr\" (UID: \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\") " pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:16.388947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.388799    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9235dbab-c8e3-4786-8789-46f194cd9937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6\" (UID: \"9235dbab-c8e3-4786-8789-46f194cd9937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:16.388947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.388829    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxfr\" (UniqueName: \"kubernetes.io/projected/9235dbab-c8e3-4786-8789-46f194cd9937-kube-api-access-qbxfr\") pod \"payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6\" (UID: \"9235dbab-c8e3-4786-8789-46f194cd9937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:16.388947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.388863    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-plugins-config-volume\") pod \"payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr\" (UID: \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\") " pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:16.389657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.389636    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-plugins-config-volume\") pod \"payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr\" (UID: \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\") " pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:16.389749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.389726    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9235dbab-c8e3-4786-8789-46f194cd9937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6\" (UID: \"9235dbab-c8e3-4786-8789-46f194cd9937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:16.397348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.397320    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxq6w\" (UniqueName: \"kubernetes.io/projected/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-kube-api-access-hxq6w\") pod \"payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr\" (UID: \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\") " pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:16.397497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.397475    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxfr\" (UniqueName: \"kubernetes.io/projected/9235dbab-c8e3-4786-8789-46f194cd9937-kube-api-access-qbxfr\") pod \"payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6\" (UID: \"9235dbab-c8e3-4786-8789-46f194cd9937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:16.540631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.540524    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:16.548800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.548776    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:16.751377 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.751234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:16.773463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.773432    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"]
Nov 03 10:40:16.778510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.778485    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:16.785046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.784981    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-afe85416\""
Nov 03 10:40:16.786310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.786266    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-afe85416-dockercfg-zwxt9\""
Nov 03 10:40:16.787505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.787475    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"]
Nov 03 10:40:16.845872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.845793    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"]
Nov 03 10:40:16.849384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.848834    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"]
Nov 03 10:40:16.896093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.895743    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zv6w\" (UniqueName: \"kubernetes.io/projected/53eba721-0edd-49ad-ad28-7f9d874d4839-kube-api-access-9zv6w\") pod \"payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85\" (UID: \"53eba721-0edd-49ad-ad28-7f9d874d4839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:16.896093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.895824    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53eba721-0edd-49ad-ad28-7f9d874d4839-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85\" (UID: \"53eba721-0edd-49ad-ad28-7f9d874d4839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:16.904810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.904705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:16.996810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.996773    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zv6w\" (UniqueName: \"kubernetes.io/projected/53eba721-0edd-49ad-ad28-7f9d874d4839-kube-api-access-9zv6w\") pod \"payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85\" (UID: \"53eba721-0edd-49ad-ad28-7f9d874d4839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:16.996971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.996880    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53eba721-0edd-49ad-ad28-7f9d874d4839-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85\" (UID: \"53eba721-0edd-49ad-ad28-7f9d874d4839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:16.997844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:16.997719    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53eba721-0edd-49ad-ad28-7f9d874d4839-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85\" (UID: \"53eba721-0edd-49ad-ad28-7f9d874d4839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:17.005179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.005151    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zv6w\" (UniqueName: \"kubernetes.io/projected/53eba721-0edd-49ad-ad28-7f9d874d4839-kube-api-access-9zv6w\") pod \"payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85\" (UID: \"53eba721-0edd-49ad-ad28-7f9d874d4839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:17.071427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.071392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:17.099613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.099495    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:17.234293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.234269    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:17.234482 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.234380    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:17.304956 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.304917    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr" event={"ID":"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c","Type":"ContainerStarted","Data":"6b57a83714625f9e39d289be55fdbfce3ee92a0ca7a80b010292cc8c3f91b28c"}
Nov 03 10:40:17.305175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.304987    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr" event={"ID":"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c","Type":"ContainerStarted","Data":"4112f97a305e530705786151586c024275458d1d08bace836062cb58096ec3a1"}
Nov 03 10:40:17.305816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.305793    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:17.308052 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.308010    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6" event={"ID":"9235dbab-c8e3-4786-8789-46f194cd9937","Type":"ContainerStarted","Data":"bb353a674181b91c75b2e44a4d86e056cd922232fbe6e732f6f310cda591729c"}
Nov 03 10:40:17.308210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.308070    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6" event={"ID":"9235dbab-c8e3-4786-8789-46f194cd9937","Type":"ContainerStarted","Data":"7796f7b188fa53d428cebf04b307756ffafb6b229fce37f1fcc01d4f62630eb4"}
Nov 03 10:40:17.308689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.308669    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:17.326319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.326264    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr" podStartSLOduration=1.3262438539999999 podStartE2EDuration="1.326243854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:17.321328985 +0000 UTC m=+647.526911065" watchObservedRunningTime="2026-11-03 10:40:17.326243854 +0000 UTC m=+647.531825913"
Nov 03 10:40:17.337546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.337485    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6" podStartSLOduration=1.337464939 podStartE2EDuration="1.337464939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:17.336750261 +0000 UTC m=+647.542332320" watchObservedRunningTime="2026-11-03 10:40:17.337464939 +0000 UTC m=+647.543047086"
Nov 03 10:40:17.391545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.391463    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"]
Nov 03 10:40:17.854803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:17.854233    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:18.054086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:18.054054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:18.323874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:18.323829    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85" event={"ID":"53eba721-0edd-49ad-ad28-7f9d874d4839","Type":"ContainerStarted","Data":"4ee0f40bffdc30309633becc4eb94e28f1e8f78b5bbf9dfc30b730f59b08c98b"}
Nov 03 10:40:18.324446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:18.324425    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:18.324564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:18.324460    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85" event={"ID":"53eba721-0edd-49ad-ad28-7f9d874d4839","Type":"ContainerStarted","Data":"e91f6ebb35851faf19f0e7911172663b7dda8991942e12483138c11c7dec778f"}
Nov 03 10:40:18.343060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:18.342998    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85" podStartSLOduration=2.342980994 podStartE2EDuration="2.342980994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:18.339277555 +0000 UTC m=+648.544859609" watchObservedRunningTime="2026-11-03 10:40:18.342980994 +0000 UTC m=+648.548563045"
Nov 03 10:40:18.556990 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:18.556902    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:18.570244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:18.569762    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:40:19.250622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.250009    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:19.253489 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.253424    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:19.328277 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.328238    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"]
Nov 03 10:40:19.333060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.333033    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.337925 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.336566    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"]
Nov 03 10:40:19.339233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.338326    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:40:19.339233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.338956    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:40:19.416599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.410803    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:19.440312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.440272    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94626609-55f4-419e-b0c0-2d093f357318-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.440484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.440467    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94626609-55f4-419e-b0c0-2d093f357318-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.440550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.440524    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7r2\" (UniqueName: \"kubernetes.io/projected/94626609-55f4-419e-b0c0-2d093f357318-kube-api-access-7c7r2\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.541444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.541351    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94626609-55f4-419e-b0c0-2d093f357318-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.541644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.541478    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94626609-55f4-419e-b0c0-2d093f357318-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.541644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.541514    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7r2\" (UniqueName: \"kubernetes.io/projected/94626609-55f4-419e-b0c0-2d093f357318-kube-api-access-7c7r2\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.542357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.542331    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:19.542528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.542432    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94626609-55f4-419e-b0c0-2d093f357318-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.545166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.545139    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94626609-55f4-419e-b0c0-2d093f357318-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.549003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.548980    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7r2\" (UniqueName: \"kubernetes.io/projected/94626609-55f4-419e-b0c0-2d093f357318-kube-api-access-7c7r2\") pod \"maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.654141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.654100    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:19.976536 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:19.976510    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"]
Nov 03 10:40:19.978484 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:19.978452    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94626609_55f4_419e_b0c0_2d093f357318.slice/crio-7c5fd5cc6da9916b0600fe705a95177b6923b6754255d33b1d0e139c5d0e74bf WatchSource:0}: Error finding container 7c5fd5cc6da9916b0600fe705a95177b6923b6754255d33b1d0e139c5d0e74bf: Status 404 returned error can't find the container with id 7c5fd5cc6da9916b0600fe705a95177b6923b6754255d33b1d0e139c5d0e74bf
Nov 03 10:40:20.251793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.251709    5817 patch_prober.go:29] interesting pod/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:40:20.251793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.251774    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:40:20.268053 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.267193    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:20.340634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.340603    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7" event={"ID":"94626609-55f4-419e-b0c0-2d093f357318","Type":"ContainerStarted","Data":"7c5fd5cc6da9916b0600fe705a95177b6923b6754255d33b1d0e139c5d0e74bf"}
Nov 03 10:40:20.342327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.342302    5817 generic.go:356] "Generic (PLEG): container finished" podID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerID="3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242" exitCode=0
Nov 03 10:40:20.342452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.342370    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" event={"ID":"a0fec804-7104-45c8-8c9b-9a901c1505eb","Type":"ContainerDied","Data":"3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242"}
Nov 03 10:40:20.342832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.342818    5817 scope.go:122] "RemoveContainer" containerID="3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242"
Nov 03 10:40:20.344034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.344011    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:20.344118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.344083    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:20.344156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:20.344117    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:21.044068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:21.043976    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:21.366992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:21.366904    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" event={"ID":"a0fec804-7104-45c8-8c9b-9a901c1505eb","Type":"ContainerStarted","Data":"754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8"}
Nov 03 10:40:21.748596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:21.748064    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:21.757597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:21.753053    5817 patch_prober.go:29] interesting pod/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 10:40:21.757597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:21.753109    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 10:40:21.856610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:21.855596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:22.558398 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:22.558325    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:40:22.747861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:22.744674    5817 patch_prober.go:29] interesting pod/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 10:40:22.747861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:22.744731    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 10:40:22.953599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:22.951927    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:23.744022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:23.741902    5817 patch_prober.go:29] interesting pod/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 10:40:23.744022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:23.741961    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 10:40:24.282391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:24.282360    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:24.731940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:24.731899    5817 patch_prober.go:29] interesting pod/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 10:40:24.732121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:24.732005    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 10:40:24.780721 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:24.780683    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:24.927850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:24.927728    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:24.936548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:24.936516    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"]
Nov 03 10:40:24.943263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:24.943190    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:24.947317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:24.947291    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"]
Nov 03 10:40:25.042644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.042006    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk89j\" (UniqueName: \"kubernetes.io/projected/16395492-02b6-4294-bdcf-f3f8178461fc-kube-api-access-nk89j\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.042644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.042179    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.042644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.042211    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.142955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.142922    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk89j\" (UniqueName: \"kubernetes.io/projected/16395492-02b6-4294-bdcf-f3f8178461fc-kube-api-access-nk89j\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.143091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.143021    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.143091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.143040    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.143897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.143831    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.145756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.145732    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.151213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.151183    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk89j\" (UniqueName: \"kubernetes.io/projected/16395492-02b6-4294-bdcf-f3f8178461fc-kube-api-access-nk89j\") pod \"maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.269753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.269718    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:25.311803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.310762    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:25.355458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.355414    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:25.423829 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.423790    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7" event={"ID":"94626609-55f4-419e-b0c0-2d093f357318","Type":"ContainerStarted","Data":"4d42415af7bfa58a3d79c213454d7ad6d1bdf8d71a46ec9d09bffbf376a88642"}
Nov 03 10:40:25.444593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.443955    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7" podStartSLOduration=1.585293321 podStartE2EDuration="6.443935112s" totalImagesPullingTime="4.858641791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:25.441869192 +0000 UTC m=+655.647451236" watchObservedRunningTime="2026-11-03 10:40:25.443935112 +0000 UTC m=+655.649517164"
Nov 03 10:40:25.608595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.597797    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"]
Nov 03 10:40:25.731279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.731243    5817 patch_prober.go:29] interesting pod/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 10:40:25.731450 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:25.731301    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 10:40:26.340777 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.340660    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:26.434400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.434355    5817 generic.go:356] "Generic (PLEG): container finished" podID="94626609-55f4-419e-b0c0-2d093f357318" containerID="4d42415af7bfa58a3d79c213454d7ad6d1bdf8d71a46ec9d09bffbf376a88642" exitCode=0
Nov 03 10:40:26.434595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.434522    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7" event={"ID":"94626609-55f4-419e-b0c0-2d093f357318","Type":"ContainerDied","Data":"4d42415af7bfa58a3d79c213454d7ad6d1bdf8d71a46ec9d09bffbf376a88642"}
Nov 03 10:40:26.436679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.436652    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9" event={"ID":"16395492-02b6-4294-bdcf-f3f8178461fc","Type":"ContainerStarted","Data":"6817673764fdd1f495ff48df640361025d0d36a452e128122251b1feb8bb94f2"}
Nov 03 10:40:26.436802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.436691    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9" event={"ID":"16395492-02b6-4294-bdcf-f3f8178461fc","Type":"ContainerStarted","Data":"ff01312071300ed9522df621fe1de9390be48b0815c9ac9a1b99a8301cd6bc4a"}
Nov 03 10:40:26.470826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.470769    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9" podStartSLOduration=2.470751726 podStartE2EDuration="2.470751726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:26.469112702 +0000 UTC m=+656.674694754" watchObservedRunningTime="2026-11-03 10:40:26.470751726 +0000 UTC m=+656.676333779"
Nov 03 10:40:26.842713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.842681    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:26.843534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.843510    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:26.844384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.844363    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:26.894218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.894141    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:40:26.957204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:26.957169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:27.299692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.299663    5817 patch_prober.go:29] interesting pod/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:40:27.299875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.299707    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:40:27.444873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.444834    5817 generic.go:356] "Generic (PLEG): container finished" podID="16395492-02b6-4294-bdcf-f3f8178461fc" containerID="6817673764fdd1f495ff48df640361025d0d36a452e128122251b1feb8bb94f2" exitCode=0
Nov 03 10:40:27.445248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.444949    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9" event={"ID":"16395492-02b6-4294-bdcf-f3f8178461fc","Type":"ContainerDied","Data":"6817673764fdd1f495ff48df640361025d0d36a452e128122251b1feb8bb94f2"}
Nov 03 10:40:27.621590 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.621556    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:27.673863 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.673825    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94626609-55f4-419e-b0c0-2d093f357318-kube-api-access-7c7r2\" (UniqueName: \"kubernetes.io/projected/94626609-55f4-419e-b0c0-2d093f357318-kube-api-access-7c7r2\") pod \"94626609-55f4-419e-b0c0-2d093f357318\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") "
Nov 03 10:40:27.675935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.673890    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94626609-55f4-419e-b0c0-2d093f357318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94626609-55f4-419e-b0c0-2d093f357318-maas-api-openshift-ca\") pod \"94626609-55f4-419e-b0c0-2d093f357318\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") "
Nov 03 10:40:27.675935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.674055    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94626609-55f4-419e-b0c0-2d093f357318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94626609-55f4-419e-b0c0-2d093f357318-maas-api-xks-ca\") pod \"94626609-55f4-419e-b0c0-2d093f357318\" (UID: \"94626609-55f4-419e-b0c0-2d093f357318\") "
Nov 03 10:40:27.675935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.674431    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94626609-55f4-419e-b0c0-2d093f357318-maas-api-openshift-ca" pod "94626609-55f4-419e-b0c0-2d093f357318" (UID: "94626609-55f4-419e-b0c0-2d093f357318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:27.676753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.676727    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94626609-55f4-419e-b0c0-2d093f357318-kube-api-access-7c7r2" pod "94626609-55f4-419e-b0c0-2d093f357318" (UID: "94626609-55f4-419e-b0c0-2d093f357318"). InnerVolumeSpecName "kube-api-access-7c7r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:27.676991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.676964    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94626609-55f4-419e-b0c0-2d093f357318-maas-api-xks-ca" pod "94626609-55f4-419e-b0c0-2d093f357318" (UID: "94626609-55f4-419e-b0c0-2d093f357318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:27.740927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.739114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:27.775028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.775000    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7r2\" (UniqueName: \"kubernetes.io/projected/94626609-55f4-419e-b0c0-2d093f357318-kube-api-access-7c7r2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:27.775028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.775026    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94626609-55f4-419e-b0c0-2d093f357318-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:27.775228 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:27.775037    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94626609-55f4-419e-b0c0-2d093f357318-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:28.453893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.453863    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"
Nov 03 10:40:28.454386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.453909    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7" event={"ID":"94626609-55f4-419e-b0c0-2d093f357318","Type":"ContainerDied","Data":"7c5fd5cc6da9916b0600fe705a95177b6923b6754255d33b1d0e139c5d0e74bf"}
Nov 03 10:40:28.454386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.453959    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c5fd5cc6da9916b0600fe705a95177b6923b6754255d33b1d0e139c5d0e74bf"
Nov 03 10:40:28.455772 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.455748    5817 generic.go:356] "Generic (PLEG): container finished" podID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerID="558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826" exitCode=0
Nov 03 10:40:28.455921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.455841    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" event={"ID":"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8","Type":"ContainerDied","Data":"558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826"}
Nov 03 10:40:28.456411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.456382    5817 scope.go:122] "RemoveContainer" containerID="558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826"
Nov 03 10:40:28.457818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.457790    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:28.457922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.457861    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:28.457922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.457894    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:28.618474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.618264    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:28.665652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.665625    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:28.787040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.786959    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-xks-ca\") pod \"16395492-02b6-4294-bdcf-f3f8178461fc\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") "
Nov 03 10:40:28.787040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.787007    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-openshift-ca\") pod \"16395492-02b6-4294-bdcf-f3f8178461fc\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") "
Nov 03 10:40:28.787313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.787075    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16395492-02b6-4294-bdcf-f3f8178461fc-kube-api-access-nk89j\" (UniqueName: \"kubernetes.io/projected/16395492-02b6-4294-bdcf-f3f8178461fc-kube-api-access-nk89j\") pod \"16395492-02b6-4294-bdcf-f3f8178461fc\" (UID: \"16395492-02b6-4294-bdcf-f3f8178461fc\") "
Nov 03 10:40:28.787454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.787425    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-openshift-ca" pod "16395492-02b6-4294-bdcf-f3f8178461fc" (UID: "16395492-02b6-4294-bdcf-f3f8178461fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:28.789290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.789259    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-xks-ca" pod "16395492-02b6-4294-bdcf-f3f8178461fc" (UID: "16395492-02b6-4294-bdcf-f3f8178461fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:28.789471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.789451    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16395492-02b6-4294-bdcf-f3f8178461fc-kube-api-access-nk89j" pod "16395492-02b6-4294-bdcf-f3f8178461fc" (UID: "16395492-02b6-4294-bdcf-f3f8178461fc"). InnerVolumeSpecName "kube-api-access-nk89j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:28.888338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.888297    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:28.888542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.888399    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk89j\" (UniqueName: \"kubernetes.io/projected/16395492-02b6-4294-bdcf-f3f8178461fc-kube-api-access-nk89j\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:28.888542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:28.888424    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16395492-02b6-4294-bdcf-f3f8178461fc-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:29.337612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.337585    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:29.337803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.337750    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:29.349378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.349335    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:29.457705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.457661    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"]
Nov 03 10:40:29.464429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.464402    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" event={"ID":"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8","Type":"ContainerStarted","Data":"4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653"}
Nov 03 10:40:29.465832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.465808    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"
Nov 03 10:40:29.465960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.465845    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9" event={"ID":"16395492-02b6-4294-bdcf-f3f8178461fc","Type":"ContainerDied","Data":"ff01312071300ed9522df621fe1de9390be48b0815c9ac9a1b99a8301cd6bc4a"}
Nov 03 10:40:29.465960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.465884    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff01312071300ed9522df621fe1de9390be48b0815c9ac9a1b99a8301cd6bc4a"
Nov 03 10:40:29.476313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.476286    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"]
Nov 03 10:40:29.577962 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.577917    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:29.735610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.733537    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:29.781644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.781611    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:29.781859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.781779    5817 patch_prober.go:29] interesting pod/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:40:29.781859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.781844    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 10:40:29.822598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.822420    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"]
Nov 03 10:40:29.822785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.822663    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9" podUID="e4dd2013-aa16-4a4c-920c-7a8795a549ae" containerName="maas-api" containerID="cri-o://0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4" gracePeriod=30
Nov 03 10:40:29.918133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:29.918059    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:30.141140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.141028    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:30.176771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.176735    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:30.237867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.237833    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-metrics-certs\") pod \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") "
Nov 03 10:40:30.238029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.237898    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-maas-api-tls\") pod \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") "
Nov 03 10:40:30.238029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.237967    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4dd2013-aa16-4a4c-920c-7a8795a549ae-kube-api-access-rwk7m\" (UniqueName: \"kubernetes.io/projected/e4dd2013-aa16-4a4c-920c-7a8795a549ae-kube-api-access-rwk7m\") pod \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") "
Nov 03 10:40:30.238029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.238005    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4dd2013-aa16-4a4c-920c-7a8795a549ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd2013-aa16-4a4c-920c-7a8795a549ae-openshift-service-ca\") pod \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\" (UID: \"e4dd2013-aa16-4a4c-920c-7a8795a549ae\") "
Nov 03 10:40:30.238720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.238694    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dd2013-aa16-4a4c-920c-7a8795a549ae-openshift-service-ca" pod "e4dd2013-aa16-4a4c-920c-7a8795a549ae" (UID: "e4dd2013-aa16-4a4c-920c-7a8795a549ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:30.239808 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.239547    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"]
Nov 03 10:40:30.241341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.241315    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-metrics-certs" pod "e4dd2013-aa16-4a4c-920c-7a8795a549ae" (UID: "e4dd2013-aa16-4a4c-920c-7a8795a549ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:30.243316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.243283    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3baa762a-vqrs7"]
Nov 03 10:40:30.246014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.245976    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dd2013-aa16-4a4c-920c-7a8795a549ae-kube-api-access-rwk7m" pod "e4dd2013-aa16-4a4c-920c-7a8795a549ae" (UID: "e4dd2013-aa16-4a4c-920c-7a8795a549ae"). InnerVolumeSpecName "kube-api-access-rwk7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:30.249879 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.249713    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-maas-api-tls" pod "e4dd2013-aa16-4a4c-920c-7a8795a549ae" (UID: "e4dd2013-aa16-4a4c-920c-7a8795a549ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:30.285803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.285779    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"]
Nov 03 10:40:30.286271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286258    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e4dd2013-aa16-4a4c-920c-7a8795a549ae" containerName="maas-api"
Nov 03 10:40:30.286312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286273    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4dd2013-aa16-4a4c-920c-7a8795a549ae" containerName="maas-api"
Nov 03 10:40:30.286312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286299    5817 cpu_manager.go:414] "containerMap: removing container" podUID="16395492-02b6-4294-bdcf-f3f8178461fc" containerName="revoke-keys"
Nov 03 10:40:30.286312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286305    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16395492-02b6-4294-bdcf-f3f8178461fc" containerName="revoke-keys"
Nov 03 10:40:30.286406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286319    5817 cpu_manager.go:414] "containerMap: removing container" podUID="94626609-55f4-419e-b0c0-2d093f357318" containerName="revoke-keys"
Nov 03 10:40:30.286406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286324    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94626609-55f4-419e-b0c0-2d093f357318" containerName="revoke-keys"
Nov 03 10:40:30.286406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286390    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="16395492-02b6-4294-bdcf-f3f8178461fc" containerName="revoke-keys"
Nov 03 10:40:30.286406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286399    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="94626609-55f4-419e-b0c0-2d093f357318" containerName="revoke-keys"
Nov 03 10:40:30.286525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.286409    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4dd2013-aa16-4a4c-920c-7a8795a549ae" containerName="maas-api"
Nov 03 10:40:30.327473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.327396    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"]
Nov 03 10:40:30.327809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.327788    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.330603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.330556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:30.331865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.331849    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:40:30.331982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.331957    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:40:30.339626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.339605    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwk7m\" (UniqueName: \"kubernetes.io/projected/e4dd2013-aa16-4a4c-920c-7a8795a549ae-kube-api-access-rwk7m\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:30.339693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.339628    5817 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd2013-aa16-4a4c-920c-7a8795a549ae-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:30.339693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.339638    5817 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:30.339693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.339647    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd2013-aa16-4a4c-920c-7a8795a549ae-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:30.345399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.345375    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:30.440480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.440446    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.440698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.440485    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.440698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.440594    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4k28\" (UniqueName: \"kubernetes.io/projected/c69f26cf-f1a9-4959-9d12-d82f71f40658-kube-api-access-b4k28\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.474103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.474065    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:30.474485 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.474127    5817 generic.go:356] "Generic (PLEG): container finished" podID="e4dd2013-aa16-4a4c-920c-7a8795a549ae" containerID="0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4" exitCode=0
Nov 03 10:40:30.474657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.474147    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9" event={"ID":"e4dd2013-aa16-4a4c-920c-7a8795a549ae","Type":"ContainerDied","Data":"0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4"}
Nov 03 10:40:30.474713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.474179    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"
Nov 03 10:40:30.474713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.474677    5817 scope.go:122] "RemoveContainer" containerID="0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4"
Nov 03 10:40:30.474792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.474658    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9" event={"ID":"e4dd2013-aa16-4a4c-920c-7a8795a549ae","Type":"ContainerDied","Data":"cffab71850d11d50aac955f0f7f72355f1b06ac1fcf0464222886bd70913c01d"}
Nov 03 10:40:30.486425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.486409    5817 scope.go:122] "RemoveContainer" containerID="0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4"
Nov 03 10:40:30.528703 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:30.528663    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4\": container with ID starting with 0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4 not found: ID does not exist" containerID="0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4"
Nov 03 10:40:30.528977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.528947    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4"} err="failed to get container status \"0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4\": rpc error: code = NotFound desc = could not find container \"0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4\": container with ID starting with 0f6d335d179815b6953afa98f67521d01b0b5713faba8fcb33badcab2ae160b4 not found: ID does not exist"
Nov 03 10:40:30.548157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.548113    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4k28\" (UniqueName: \"kubernetes.io/projected/c69f26cf-f1a9-4959-9d12-d82f71f40658-kube-api-access-b4k28\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.548354 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.548307    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.548354 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.548337    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.549529 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.549480    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.552236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.552212    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"]
Nov 03 10:40:30.553205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.553165    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.555216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.555193    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3baa762a-6ff676d5c-tdnb9"]
Nov 03 10:40:30.556161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.556127    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4k28\" (UniqueName: \"kubernetes.io/projected/c69f26cf-f1a9-4959-9d12-d82f71f40658-kube-api-access-b4k28\") pod \"maas-api-revoke-keys-e2e-ait-19708dae-5ccqd\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.613206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.613115    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:30.643972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.643943    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:30.782135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.782108    5817 patch_prober.go:29] interesting pod/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:40:30.782292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.782160    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 10:40:30.957057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:30.957028    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"]
Nov 03 10:40:31.183931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.183897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:31.316357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.314174    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"]
Nov 03 10:40:31.323325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.322841    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"]
Nov 03 10:40:31.323325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.322955    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.335384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.335354    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:31.457532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.457496    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvfv\" (UniqueName: \"kubernetes.io/projected/6c474493-16b5-4db4-ae2e-cae433ba058f-kube-api-access-vwvfv\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.457723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.457672    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.457835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.457815    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.484106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.484022    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd" event={"ID":"c69f26cf-f1a9-4959-9d12-d82f71f40658","Type":"ContainerStarted","Data":"1bff1b19fb85fdbe38fc3ebb61ff999a12da6406b305a6b4117c62d14228cd77"}
Nov 03 10:40:31.484535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.484115    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd" event={"ID":"c69f26cf-f1a9-4959-9d12-d82f71f40658","Type":"ContainerStarted","Data":"f1582f4a9b5accb8016cbe7f87b4e0ec1caf3d4773abc09da8dcf0b687353a5b"}
Nov 03 10:40:31.521222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.521159    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd" podStartSLOduration=1.521137921 podStartE2EDuration="1.521137921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:31.519917683 +0000 UTC m=+661.725499734" watchObservedRunningTime="2026-11-03 10:40:31.521137921 +0000 UTC m=+661.726719973"
Nov 03 10:40:31.559375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.559339    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.559375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.559378    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvfv\" (UniqueName: \"kubernetes.io/projected/6c474493-16b5-4db4-ae2e-cae433ba058f-kube-api-access-vwvfv\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.559558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.559439    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.560176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.560156    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.562214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.562196    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.569869 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.569847    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvfv\" (UniqueName: \"kubernetes.io/projected/6c474493-16b5-4db4-ae2e-cae433ba058f-kube-api-access-vwvfv\") pod \"maas-api-revoke-keys-e2e-ait-afe85416-wr9l5\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.636730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.636698    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:31.652597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.652551    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"]
Nov 03 10:40:31.657692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.657670    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.665055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.664661    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b1c6f6-openshift-default-dockercfg-h8q5b\""
Nov 03 10:40:31.668772 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.668749    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"]
Nov 03 10:40:31.759603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.746890    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763134    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763231    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763269    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763294    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmp2s\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-kube-api-access-tmp2s\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763332    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763386    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763454    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763618    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.767594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.763683    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235f4586-fb1b-400d-9a25-fb73a5aa5630-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.789599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.781180    5817 patch_prober.go:29] interesting pod/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:40:31.789599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.781258    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 10:40:31.864827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.864687    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.864827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.864726    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.864827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.864765    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235f4586-fb1b-400d-9a25-fb73a5aa5630-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.864827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.864816    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865084 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.864882    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865084 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.864914    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.865130    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.865178    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmp2s\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-kube-api-access-tmp2s\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.865178    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.865222    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.865309    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.865667    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.865829    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235f4586-fb1b-400d-9a25-fb73a5aa5630-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.865846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.865837    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.867669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.867647    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.868520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.868474    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.872821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.872781    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.873063 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.873045    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmp2s\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-kube-api-access-tmp2s\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:31.941988 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.941951    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94626609-55f4-419e-b0c0-2d093f357318" path="/var/lib/kubelet/pods/94626609-55f4-419e-b0c0-2d093f357318/volumes"
Nov 03 10:40:31.948353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.948158    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4dd2013-aa16-4a4c-920c-7a8795a549ae" path="/var/lib/kubelet/pods/e4dd2013-aa16-4a4c-920c-7a8795a549ae/volumes"
Nov 03 10:40:31.957025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.956992    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:31.960953 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.960931    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"]
Nov 03 10:40:31.962423 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:31.962393    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c474493_16b5_4db4_ae2e_cae433ba058f.slice/crio-b1d62845d52d618e63b7b9b32fc031190f71b6bdc58f680d15ce301bd8b1d19b WatchSource:0}: Error finding container b1d62845d52d618e63b7b9b32fc031190f71b6bdc58f680d15ce301bd8b1d19b: Status 404 returned error can't find the container with id b1d62845d52d618e63b7b9b32fc031190f71b6bdc58f680d15ce301bd8b1d19b
Nov 03 10:40:31.977886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:31.977868    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:32.225083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.224997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:32.318635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.316175    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:40:32.420439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.420413    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"]
Nov 03 10:40:32.420599 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:32.420417    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235f4586_fb1b_400d_9a25_fb73a5aa5630.slice/crio-7853902f9160b3bb567a9c01504c3d12304b1633770e9282ba0c0cfa364e60ea WatchSource:0}: Error finding container 7853902f9160b3bb567a9c01504c3d12304b1633770e9282ba0c0cfa364e60ea: Status 404 returned error can't find the container with id 7853902f9160b3bb567a9c01504c3d12304b1633770e9282ba0c0cfa364e60ea
Nov 03 10:40:32.422700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.422668    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:32.422811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.422772    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:32.422871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.422821    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:32.520168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.519610    5817 generic.go:356] "Generic (PLEG): container finished" podID="c69f26cf-f1a9-4959-9d12-d82f71f40658" containerID="1bff1b19fb85fdbe38fc3ebb61ff999a12da6406b305a6b4117c62d14228cd77" exitCode=0
Nov 03 10:40:32.520168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.519742    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd" event={"ID":"c69f26cf-f1a9-4959-9d12-d82f71f40658","Type":"ContainerDied","Data":"1bff1b19fb85fdbe38fc3ebb61ff999a12da6406b305a6b4117c62d14228cd77"}
Nov 03 10:40:32.523786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.523088    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5" event={"ID":"6c474493-16b5-4db4-ae2e-cae433ba058f","Type":"ContainerStarted","Data":"f8079976cf57191b50a3d36fdd0e1695dff542912be4b5cb51c38669388fc6d6"}
Nov 03 10:40:32.523786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.523144    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5" event={"ID":"6c474493-16b5-4db4-ae2e-cae433ba058f","Type":"ContainerStarted","Data":"b1d62845d52d618e63b7b9b32fc031190f71b6bdc58f680d15ce301bd8b1d19b"}
Nov 03 10:40:32.525961 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.525939    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" event={"ID":"235f4586-fb1b-400d-9a25-fb73a5aa5630","Type":"ContainerStarted","Data":"7853902f9160b3bb567a9c01504c3d12304b1633770e9282ba0c0cfa364e60ea"}
Nov 03 10:40:32.553640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.553563    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5" podStartSLOduration=1.553549742 podStartE2EDuration="1.553549742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:32.551746967 +0000 UTC m=+662.757329017" watchObservedRunningTime="2026-11-03 10:40:32.553549742 +0000 UTC m=+662.759131804"
Nov 03 10:40:32.782080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.782046    5817 patch_prober.go:29] interesting pod/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:40:32.782271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:32.782103    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 10:40:33.044674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.044092    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:33.534916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.534880    5817 generic.go:356] "Generic (PLEG): container finished" podID="6c474493-16b5-4db4-ae2e-cae433ba058f" containerID="f8079976cf57191b50a3d36fdd0e1695dff542912be4b5cb51c38669388fc6d6" exitCode=0
Nov 03 10:40:33.535352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.535027    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5" event={"ID":"6c474493-16b5-4db4-ae2e-cae433ba058f","Type":"ContainerDied","Data":"f8079976cf57191b50a3d36fdd0e1695dff542912be4b5cb51c38669388fc6d6"}
Nov 03 10:40:33.536720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.536693    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" event={"ID":"235f4586-fb1b-400d-9a25-fb73a5aa5630","Type":"ContainerStarted","Data":"adf053cb278ded75802f7c25edd396ee1024a08f3c1991b88479071bbea0d83c"}
Nov 03 10:40:33.572735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.570933    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podStartSLOduration=2.5709177 podStartE2EDuration="2.5709177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:33.569253158 +0000 UTC m=+663.774835208" watchObservedRunningTime="2026-11-03 10:40:33.5709177 +0000 UTC m=+663.776499797"
Nov 03 10:40:33.746678 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.746649    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:33.772466 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.772433    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:33.781839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.781338    5817 patch_prober.go:29] interesting pod/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:40:33.781839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.781400    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 10:40:33.784724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.784668    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69f26cf-f1a9-4959-9d12-d82f71f40658-kube-api-access-b4k28\" (UniqueName: \"kubernetes.io/projected/c69f26cf-f1a9-4959-9d12-d82f71f40658-kube-api-access-b4k28\") pod \"c69f26cf-f1a9-4959-9d12-d82f71f40658\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") "
Nov 03 10:40:33.784873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.784785    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-xks-ca\") pod \"c69f26cf-f1a9-4959-9d12-d82f71f40658\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") "
Nov 03 10:40:33.784976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.784927    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-openshift-ca\") pod \"c69f26cf-f1a9-4959-9d12-d82f71f40658\" (UID: \"c69f26cf-f1a9-4959-9d12-d82f71f40658\") "
Nov 03 10:40:33.785588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.785551    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-openshift-ca" pod "c69f26cf-f1a9-4959-9d12-d82f71f40658" (UID: "c69f26cf-f1a9-4959-9d12-d82f71f40658"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:33.814923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.813936    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69f26cf-f1a9-4959-9d12-d82f71f40658-kube-api-access-b4k28" pod "c69f26cf-f1a9-4959-9d12-d82f71f40658" (UID: "c69f26cf-f1a9-4959-9d12-d82f71f40658"). InnerVolumeSpecName "kube-api-access-b4k28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:33.814923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.814021    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-xks-ca" pod "c69f26cf-f1a9-4959-9d12-d82f71f40658" (UID: "c69f26cf-f1a9-4959-9d12-d82f71f40658"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:33.892595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.886640    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4k28\" (UniqueName: \"kubernetes.io/projected/c69f26cf-f1a9-4959-9d12-d82f71f40658-kube-api-access-b4k28\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:33.892595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.886677    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:33.892595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.886692    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69f26cf-f1a9-4959-9d12-d82f71f40658-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:33.979140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.979100    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:33.984975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:33.984950    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:34.458630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.458589    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j" podUID="3a32d224-89de-4f3c-9230-cdbd76db9d60" containerName="payload-processing" containerID="cri-o://c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118" gracePeriod=25
Nov 03 10:40:34.477452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.477417    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g" podUID="3e5226b0-0969-41f1-99b0-308e930c1211" containerName="payload-pre-processing" containerID="cri-o://8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5" gracePeriod=25
Nov 03 10:40:34.543732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.543700    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"
Nov 03 10:40:34.544234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.544213    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd" event={"ID":"c69f26cf-f1a9-4959-9d12-d82f71f40658","Type":"ContainerDied","Data":"f1582f4a9b5accb8016cbe7f87b4e0ec1caf3d4773abc09da8dcf0b687353a5b"}
Nov 03 10:40:34.544316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.544260    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1582f4a9b5accb8016cbe7f87b4e0ec1caf3d4773abc09da8dcf0b687353a5b"
Nov 03 10:40:34.545045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.545025    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:34.545887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.545870    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:34.566285 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.566040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:34.781946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.781913    5817 patch_prober.go:29] interesting pod/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:40:34.782118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.781980    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 10:40:34.890757 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.890724    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:34.893919 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.893898    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:34.897760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:34.897741    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:35.023950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.023868    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5226b0-0969-41f1-99b0-308e930c1211-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5226b0-0969-41f1-99b0-308e930c1211-plugins-config-volume\") pod \"3e5226b0-0969-41f1-99b0-308e930c1211\" (UID: \"3e5226b0-0969-41f1-99b0-308e930c1211\") "
Nov 03 10:40:35.023950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.023912    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a32d224-89de-4f3c-9230-cdbd76db9d60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a32d224-89de-4f3c-9230-cdbd76db9d60-plugins-config-volume\") pod \"3a32d224-89de-4f3c-9230-cdbd76db9d60\" (UID: \"3a32d224-89de-4f3c-9230-cdbd76db9d60\") "
Nov 03 10:40:35.024175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024051    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c474493-16b5-4db4-ae2e-cae433ba058f-kube-api-access-vwvfv\" (UniqueName: \"kubernetes.io/projected/6c474493-16b5-4db4-ae2e-cae433ba058f-kube-api-access-vwvfv\") pod \"6c474493-16b5-4db4-ae2e-cae433ba058f\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") "
Nov 03 10:40:35.024175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024138    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5226b0-0969-41f1-99b0-308e930c1211-kube-api-access-qfjdj\" (UniqueName: \"kubernetes.io/projected/3e5226b0-0969-41f1-99b0-308e930c1211-kube-api-access-qfjdj\") pod \"3e5226b0-0969-41f1-99b0-308e930c1211\" (UID: \"3e5226b0-0969-41f1-99b0-308e930c1211\") "
Nov 03 10:40:35.024282 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024187    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a32d224-89de-4f3c-9230-cdbd76db9d60-kube-api-access-fzhdf\" (UniqueName: \"kubernetes.io/projected/3a32d224-89de-4f3c-9230-cdbd76db9d60-kube-api-access-fzhdf\") pod \"3a32d224-89de-4f3c-9230-cdbd76db9d60\" (UID: \"3a32d224-89de-4f3c-9230-cdbd76db9d60\") "
Nov 03 10:40:35.024343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024281    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-xks-ca\") pod \"6c474493-16b5-4db4-ae2e-cae433ba058f\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") "
Nov 03 10:40:35.024343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024320    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-openshift-ca\") pod \"6c474493-16b5-4db4-ae2e-cae433ba058f\" (UID: \"6c474493-16b5-4db4-ae2e-cae433ba058f\") "
Nov 03 10:40:35.024343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024331    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5226b0-0969-41f1-99b0-308e930c1211-plugins-config-volume" pod "3e5226b0-0969-41f1-99b0-308e930c1211" (UID: "3e5226b0-0969-41f1-99b0-308e930c1211"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:35.024487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024334    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a32d224-89de-4f3c-9230-cdbd76db9d60-plugins-config-volume" pod "3a32d224-89de-4f3c-9230-cdbd76db9d60" (UID: "3a32d224-89de-4f3c-9230-cdbd76db9d60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:35.024803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024744    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5226b0-0969-41f1-99b0-308e930c1211-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:35.024803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024769    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a32d224-89de-4f3c-9230-cdbd76db9d60-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:35.024803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.024827    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-openshift-ca" pod "6c474493-16b5-4db4-ae2e-cae433ba058f" (UID: "6c474493-16b5-4db4-ae2e-cae433ba058f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:35.026507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.026484    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c474493-16b5-4db4-ae2e-cae433ba058f-kube-api-access-vwvfv" pod "6c474493-16b5-4db4-ae2e-cae433ba058f" (UID: "6c474493-16b5-4db4-ae2e-cae433ba058f"). InnerVolumeSpecName "kube-api-access-vwvfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:35.026839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.026811    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-xks-ca" pod "6c474493-16b5-4db4-ae2e-cae433ba058f" (UID: "6c474493-16b5-4db4-ae2e-cae433ba058f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:35.027159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.027128    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a32d224-89de-4f3c-9230-cdbd76db9d60-kube-api-access-fzhdf" pod "3a32d224-89de-4f3c-9230-cdbd76db9d60" (UID: "3a32d224-89de-4f3c-9230-cdbd76db9d60"). InnerVolumeSpecName "kube-api-access-fzhdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:35.027268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.027158    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5226b0-0969-41f1-99b0-308e930c1211-kube-api-access-qfjdj" pod "3e5226b0-0969-41f1-99b0-308e930c1211" (UID: "3e5226b0-0969-41f1-99b0-308e930c1211"). InnerVolumeSpecName "kube-api-access-qfjdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:35.122026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.121990    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"]
Nov 03 10:40:35.125349 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.125319    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:35.125349 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.125350    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c474493-16b5-4db4-ae2e-cae433ba058f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:35.125556 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.125363    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwvfv\" (UniqueName: \"kubernetes.io/projected/6c474493-16b5-4db4-ae2e-cae433ba058f-kube-api-access-vwvfv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:35.125556 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.125372    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjdj\" (UniqueName: \"kubernetes.io/projected/3e5226b0-0969-41f1-99b0-308e930c1211-kube-api-access-qfjdj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:35.125556 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.125381    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzhdf\" (UniqueName: \"kubernetes.io/projected/3a32d224-89de-4f3c-9230-cdbd76db9d60-kube-api-access-fzhdf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:35.234239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.234198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:35.369109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.369027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:35.550184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.550155    5817 generic.go:356] "Generic (PLEG): container finished" podID="3e5226b0-0969-41f1-99b0-308e930c1211" containerID="8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5" exitCode=0
Nov 03 10:40:35.550671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.550200    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"
Nov 03 10:40:35.550671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.550230    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g" event={"ID":"3e5226b0-0969-41f1-99b0-308e930c1211","Type":"ContainerDied","Data":"8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5"}
Nov 03 10:40:35.550671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.550276    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g" event={"ID":"3e5226b0-0969-41f1-99b0-308e930c1211","Type":"ContainerDied","Data":"ef2377d4ece2d20486ec202bf4a6aab9b77ea65a90a563bfc213acd7d978d4f1"}
Nov 03 10:40:35.550671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.550297    5817 scope.go:122] "RemoveContainer" containerID="8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5"
Nov 03 10:40:35.551731 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.551706    5817 generic.go:356] "Generic (PLEG): container finished" podID="3a32d224-89de-4f3c-9230-cdbd76db9d60" containerID="c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118" exitCode=0
Nov 03 10:40:35.551846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.551746    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"
Nov 03 10:40:35.551846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.551802    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j" event={"ID":"3a32d224-89de-4f3c-9230-cdbd76db9d60","Type":"ContainerDied","Data":"c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118"}
Nov 03 10:40:35.551945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.551864    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j" event={"ID":"3a32d224-89de-4f3c-9230-cdbd76db9d60","Type":"ContainerDied","Data":"0ca9677bb58ecace344241d722a3cb2e9d1fe893b3a1d13d0c1c25794811111e"}
Nov 03 10:40:35.557844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.557825    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"
Nov 03 10:40:35.557959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.557828    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5" event={"ID":"6c474493-16b5-4db4-ae2e-cae433ba058f","Type":"ContainerDied","Data":"b1d62845d52d618e63b7b9b32fc031190f71b6bdc58f680d15ce301bd8b1d19b"}
Nov 03 10:40:35.557959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.557951    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1d62845d52d618e63b7b9b32fc031190f71b6bdc58f680d15ce301bd8b1d19b"
Nov 03 10:40:35.562185 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.562169    5817 scope.go:122] "RemoveContainer" containerID="8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5"
Nov 03 10:40:35.562476 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:35.562456    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5\": container with ID starting with 8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5 not found: ID does not exist" containerID="8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5"
Nov 03 10:40:35.562609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.562489    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5"} err="failed to get container status \"8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5\": rpc error: code = NotFound desc = could not find container \"8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5\": container with ID starting with 8c4f47dd564a6469c15e567f5656b5c3e9c32ff52db6f174f8533fe1f65ab7d5 not found: ID does not exist"
Nov 03 10:40:35.562609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.562513    5817 scope.go:122] "RemoveContainer" containerID="c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118"
Nov 03 10:40:35.572305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.572289    5817 scope.go:122] "RemoveContainer" containerID="c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118"
Nov 03 10:40:35.573355 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:35.573329    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118\": container with ID starting with c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118 not found: ID does not exist" containerID="c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118"
Nov 03 10:40:35.573457 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.573366    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118"} err="failed to get container status \"c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118\": rpc error: code = NotFound desc = could not find container \"c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118\": container with ID starting with c102fc1e59329319dd7525d1757d2c922071838ba41a4669d7d9ea8ed3d6a118 not found: ID does not exist"
Nov 03 10:40:35.575994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.575975    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"]
Nov 03 10:40:35.578325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.578307    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3baa762a-d4698489f-f7t9g"]
Nov 03 10:40:35.586618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.586595    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"]
Nov 03 10:40:35.588931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.588914    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3baa762a-6bb7f579f5-wgq9j"]
Nov 03 10:40:35.713264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.713238    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"]
Nov 03 10:40:35.726839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.726807    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"]
Nov 03 10:40:35.785437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.785407    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:35.785845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.785824    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:35.787098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.787057    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:35.870954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.870671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:35.901921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.901892    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a32d224-89de-4f3c-9230-cdbd76db9d60" path="/var/lib/kubelet/pods/3a32d224-89de-4f3c-9230-cdbd76db9d60/volumes"
Nov 03 10:40:35.902420 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:35.902399    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5226b0-0969-41f1-99b0-308e930c1211" path="/var/lib/kubelet/pods/3e5226b0-0969-41f1-99b0-308e930c1211/volumes"
Nov 03 10:40:36.045398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:36.045313    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:36.449137 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:36.449097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:37.180092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.180045    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:37.202824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.202790    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"]
Nov 03 10:40:37.203427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203405    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3e5226b0-0969-41f1-99b0-308e930c1211" containerName="payload-pre-processing"
Nov 03 10:40:37.203427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203426    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5226b0-0969-41f1-99b0-308e930c1211" containerName="payload-pre-processing"
Nov 03 10:40:37.203652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203478    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3a32d224-89de-4f3c-9230-cdbd76db9d60" containerName="payload-processing"
Nov 03 10:40:37.203652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203485    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a32d224-89de-4f3c-9230-cdbd76db9d60" containerName="payload-processing"
Nov 03 10:40:37.203652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203507    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c69f26cf-f1a9-4959-9d12-d82f71f40658" containerName="revoke-keys"
Nov 03 10:40:37.203652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203514    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69f26cf-f1a9-4959-9d12-d82f71f40658" containerName="revoke-keys"
Nov 03 10:40:37.203652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203530    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6c474493-16b5-4db4-ae2e-cae433ba058f" containerName="revoke-keys"
Nov 03 10:40:37.203652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203537    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c474493-16b5-4db4-ae2e-cae433ba058f" containerName="revoke-keys"
Nov 03 10:40:37.203936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203656    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a32d224-89de-4f3c-9230-cdbd76db9d60" containerName="payload-processing"
Nov 03 10:40:37.203936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203672    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69f26cf-f1a9-4959-9d12-d82f71f40658" containerName="revoke-keys"
Nov 03 10:40:37.203936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203683    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c474493-16b5-4db4-ae2e-cae433ba058f" containerName="revoke-keys"
Nov 03 10:40:37.203936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.203692    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5226b0-0969-41f1-99b0-308e930c1211" containerName="payload-pre-processing"
Nov 03 10:40:37.208682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.208659    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:37.210320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.210298    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"]
Nov 03 10:40:37.213868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.213849    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:37.221850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.221779    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"]
Nov 03 10:40:37.223448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.223416    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b1c6f6\""
Nov 03 10:40:37.240209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.240180    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"]
Nov 03 10:40:37.351206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.351163    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b827954-c908-4603-998d-501f19de5a7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv\" (UID: \"6b827954-c908-4603-998d-501f19de5a7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:37.351409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.351254    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246w5\" (UniqueName: \"kubernetes.io/projected/6b827954-c908-4603-998d-501f19de5a7a-kube-api-access-246w5\") pod \"payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv\" (UID: \"6b827954-c908-4603-998d-501f19de5a7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:37.351409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.351308    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh7g6\" (UniqueName: \"kubernetes.io/projected/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-kube-api-access-rh7g6\") pod \"payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9\" (UID: \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:37.351409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.351350    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9\" (UID: \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:37.371590 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.359444    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:37.452752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.452652    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246w5\" (UniqueName: \"kubernetes.io/projected/6b827954-c908-4603-998d-501f19de5a7a-kube-api-access-246w5\") pod \"payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv\" (UID: \"6b827954-c908-4603-998d-501f19de5a7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:37.452752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.452732    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh7g6\" (UniqueName: \"kubernetes.io/projected/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-kube-api-access-rh7g6\") pod \"payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9\" (UID: \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:37.452997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.452774    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9\" (UID: \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:37.452997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.452911    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b827954-c908-4603-998d-501f19de5a7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv\" (UID: \"6b827954-c908-4603-998d-501f19de5a7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:37.453769 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.453746    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b827954-c908-4603-998d-501f19de5a7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv\" (UID: \"6b827954-c908-4603-998d-501f19de5a7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:37.453838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.453750    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9\" (UID: \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:37.460809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.460777    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-246w5\" (UniqueName: \"kubernetes.io/projected/6b827954-c908-4603-998d-501f19de5a7a-kube-api-access-246w5\") pod \"payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv\" (UID: \"6b827954-c908-4603-998d-501f19de5a7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:37.460931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.460852    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh7g6\" (UniqueName: \"kubernetes.io/projected/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-kube-api-access-rh7g6\") pod \"payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9\" (UID: \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:37.527754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.527723    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:37.534977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.534945    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:37.542194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.542162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:37.808892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.808734    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"]
Nov 03 10:40:37.811321 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:37.811296    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f5eb5a_b57d_4aa8_9354_4a25bcd3e99d.slice/crio-9fb17341791a734d873de88da63b817171661be09fc0324e79aadf1792dca58a WatchSource:0}: Error finding container 9fb17341791a734d873de88da63b817171661be09fc0324e79aadf1792dca58a: Status 404 returned error can't find the container with id 9fb17341791a734d873de88da63b817171661be09fc0324e79aadf1792dca58a
Nov 03 10:40:37.827206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:37.827176    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"]
Nov 03 10:40:37.831254 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:37.831227    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b827954_c908_4603_998d_501f19de5a7a.slice/crio-1039d975d8f69b9ad7ff14707cb97e9d0c4e853816612601d9b3fba91f0c143c WatchSource:0}: Error finding container 1039d975d8f69b9ad7ff14707cb97e9d0c4e853816612601d9b3fba91f0c143c: Status 404 returned error can't find the container with id 1039d975d8f69b9ad7ff14707cb97e9d0c4e853816612601d9b3fba91f0c143c
Nov 03 10:40:38.467732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.467696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:38.583768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.583740    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9" event={"ID":"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d","Type":"ContainerStarted","Data":"dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9"}
Nov 03 10:40:38.583918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.583786    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9" event={"ID":"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d","Type":"ContainerStarted","Data":"9fb17341791a734d873de88da63b817171661be09fc0324e79aadf1792dca58a"}
Nov 03 10:40:38.583918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.583841    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:38.585366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.585340    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv" event={"ID":"6b827954-c908-4603-998d-501f19de5a7a","Type":"ContainerStarted","Data":"a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e"}
Nov 03 10:40:38.585491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.585386    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv" event={"ID":"6b827954-c908-4603-998d-501f19de5a7a","Type":"ContainerStarted","Data":"1039d975d8f69b9ad7ff14707cb97e9d0c4e853816612601d9b3fba91f0c143c"}
Nov 03 10:40:38.585491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.585426    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:38.606811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.606766    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9" podStartSLOduration=1.6067532839999998 podStartE2EDuration="1.606753284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:38.606166608 +0000 UTC m=+668.811748676" watchObservedRunningTime="2026-11-03 10:40:38.606753284 +0000 UTC m=+668.812335334"
Nov 03 10:40:38.635162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:38.635118    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv" podStartSLOduration=1.635105601 podStartE2EDuration="1.635105601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:38.632460525 +0000 UTC m=+668.838042576" watchObservedRunningTime="2026-11-03 10:40:38.635105601 +0000 UTC m=+668.840687652"
Nov 03 10:40:39.078813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.078776    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:39.213965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.213930    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:39.262848 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.262808    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"]
Nov 03 10:40:39.267367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.267342    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.269606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.269284    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"]
Nov 03 10:40:39.275733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.275708    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a6822cf-9cnl9"]
Nov 03 10:40:39.276264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.276244    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 10:40:39.276351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.276250    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 10:40:39.278018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.277993    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 10:40:39.278790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.278587    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l8bvs\""
Nov 03 10:40:39.278790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.278629    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b1c6f6-serving-cert\""
Nov 03 10:40:39.286611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.286590    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"]
Nov 03 10:40:39.389163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.389080    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:39.391152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.391131    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.391218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.391172    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-metrics-certs\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.391289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.391270    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/812be958-9740-4509-9b3a-39b8b1cebc50-kube-api-access-kflts\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.391360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.391337    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/812be958-9740-4509-9b3a-39b8b1cebc50-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.492526 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.492497    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.492995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.492542    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-metrics-certs\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.492995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.492643    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/812be958-9740-4509-9b3a-39b8b1cebc50-kube-api-access-kflts\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.492995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.492719    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/812be958-9740-4509-9b3a-39b8b1cebc50-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.493474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.493449    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/812be958-9740-4509-9b3a-39b8b1cebc50-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.495849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.495826    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-metrics-certs\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.496402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.496380    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.501356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.501326    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/812be958-9740-4509-9b3a-39b8b1cebc50-kube-api-access-kflts\") pod \"maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.544631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.544597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:39.592357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.592329    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:39.687045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.686890    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:39.902877 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.902840    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16395492-02b6-4294-bdcf-f3f8178461fc" path="/var/lib/kubelet/pods/16395492-02b6-4294-bdcf-f3f8178461fc/volumes"
Nov 03 10:40:39.909892 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:39.909866    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812be958_9740_4509_9b3a_39b8b1cebc50.slice/crio-2d653a90856e433a96f895fdfb45f2df9ce4e3b6f7ab8d71600a307ea09a4efa WatchSource:0}: Error finding container 2d653a90856e433a96f895fdfb45f2df9ce4e3b6f7ab8d71600a307ea09a4efa: Status 404 returned error can't find the container with id 2d653a90856e433a96f895fdfb45f2df9ce4e3b6f7ab8d71600a307ea09a4efa
Nov 03 10:40:39.910518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:39.910498    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"]
Nov 03 10:40:40.122791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.122756    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb" podUID="46eaf866-9a83-4df3-950e-97299af275b5" containerName="payload-pre-processing" containerID="cri-o://1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136" gracePeriod=25
Nov 03 10:40:40.139509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.139484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:40.335979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.335951    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:40.461231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.461209    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:40.467191 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.467162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:40.492140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.492116    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"]
Nov 03 10:40:40.509206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.508964    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"]
Nov 03 10:40:40.592236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.592192    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"]
Nov 03 10:40:40.592494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.592455    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy" containerID="cri-o://4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653" gracePeriod=30
Nov 03 10:40:40.606389 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.606360    5817 generic.go:356] "Generic (PLEG): container finished" podID="46eaf866-9a83-4df3-950e-97299af275b5" containerID="1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136" exitCode=0
Nov 03 10:40:40.606510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.606474    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb" event={"ID":"46eaf866-9a83-4df3-950e-97299af275b5","Type":"ContainerDied","Data":"1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136"}
Nov 03 10:40:40.606510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.606498    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"
Nov 03 10:40:40.606693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.606530    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb" event={"ID":"46eaf866-9a83-4df3-950e-97299af275b5","Type":"ContainerDied","Data":"8ce1029bf7b7b2d01a1f2871c4ca5346df255730b79c40847c746e1c7241d141"}
Nov 03 10:40:40.606693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.606584    5817 scope.go:122] "RemoveContainer" containerID="1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136"
Nov 03 10:40:40.610057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.610029    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt" event={"ID":"812be958-9740-4509-9b3a-39b8b1cebc50","Type":"ContainerStarted","Data":"cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787"}
Nov 03 10:40:40.610159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.610076    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt" event={"ID":"812be958-9740-4509-9b3a-39b8b1cebc50","Type":"ContainerStarted","Data":"2d653a90856e433a96f895fdfb45f2df9ce4e3b6f7ab8d71600a307ea09a4efa"}
Nov 03 10:40:40.610708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.610689    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:40.611067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.611043    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:40.611492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.611468    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eaf866-9a83-4df3-950e-97299af275b5-kube-api-access-khxcq\" (UniqueName: \"kubernetes.io/projected/46eaf866-9a83-4df3-950e-97299af275b5-kube-api-access-khxcq\") pod \"46eaf866-9a83-4df3-950e-97299af275b5\" (UID: \"46eaf866-9a83-4df3-950e-97299af275b5\") "
Nov 03 10:40:40.611626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.611517    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46eaf866-9a83-4df3-950e-97299af275b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eaf866-9a83-4df3-950e-97299af275b5-plugins-config-volume\") pod \"46eaf866-9a83-4df3-950e-97299af275b5\" (UID: \"46eaf866-9a83-4df3-950e-97299af275b5\") "
Nov 03 10:40:40.612284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.612260    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46eaf866-9a83-4df3-950e-97299af275b5-plugins-config-volume" pod "46eaf866-9a83-4df3-950e-97299af275b5" (UID: "46eaf866-9a83-4df3-950e-97299af275b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:40.620888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.620856    5817 scope.go:122] "RemoveContainer" containerID="1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136"
Nov 03 10:40:40.621296 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:40.621270    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136\": container with ID starting with 1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136 not found: ID does not exist" containerID="1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136"
Nov 03 10:40:40.621385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.621303    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136"} err="failed to get container status \"1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136\": rpc error: code = NotFound desc = could not find container \"1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136\": container with ID starting with 1f300f7d599eb105329d5128bb2de2107e32f2f65576ad4f98d3527390d3f136 not found: ID does not exist"
Nov 03 10:40:40.622600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.622554    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eaf866-9a83-4df3-950e-97299af275b5-kube-api-access-khxcq" pod "46eaf866-9a83-4df3-950e-97299af275b5" (UID: "46eaf866-9a83-4df3-950e-97299af275b5"). InnerVolumeSpecName "kube-api-access-khxcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:40.641142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.641097    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt" podStartSLOduration=1.221119648 podStartE2EDuration="1.641083737s" totalImagesPullingTime="419.964089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:40.640635299 +0000 UTC m=+670.846217353" watchObservedRunningTime="2026-11-03 10:40:40.641083737 +0000 UTC m=+670.846665788"
Nov 03 10:40:40.712993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.712966    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khxcq\" (UniqueName: \"kubernetes.io/projected/46eaf866-9a83-4df3-950e-97299af275b5-kube-api-access-khxcq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:40.713166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.712998    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eaf866-9a83-4df3-950e-97299af275b5-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:40.714615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.714586    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8" podUID="80b81ee0-c86f-4ebc-94e4-686d3357642f" containerName="payload-processing" containerID="cri-o://f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8" gracePeriod=25
Nov 03 10:40:40.729684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.729658    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn" podUID="b44d293f-a431-4bd0-a278-29a79593a981" containerName="payload-pre-processing" containerID="cri-o://49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5" gracePeriod=25
Nov 03 10:40:40.750676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.750637    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:40.904008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.903972    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"]
Nov 03 10:40:40.906524 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.904206    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy" containerID="cri-o://754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8" gracePeriod=30
Nov 03 10:40:40.944507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.944179    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"]
Nov 03 10:40:40.948768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.948740    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6822cf-6bd974f755-cftjb"]
Nov 03 10:40:40.970179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:40.970114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:41.049962 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.049938    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:41.061201 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.061183    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:41.172410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.172378    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:41.219184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.219154    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44d293f-a431-4bd0-a278-29a79593a981-kube-api-access-7s5tz\" (UniqueName: \"kubernetes.io/projected/b44d293f-a431-4bd0-a278-29a79593a981-kube-api-access-7s5tz\") pod \"b44d293f-a431-4bd0-a278-29a79593a981\" (UID: \"b44d293f-a431-4bd0-a278-29a79593a981\") "
Nov 03 10:40:41.219369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.219312    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44d293f-a431-4bd0-a278-29a79593a981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44d293f-a431-4bd0-a278-29a79593a981-plugins-config-volume\") pod \"b44d293f-a431-4bd0-a278-29a79593a981\" (UID: \"b44d293f-a431-4bd0-a278-29a79593a981\") "
Nov 03 10:40:41.219369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.219343    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b81ee0-c86f-4ebc-94e4-686d3357642f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b81ee0-c86f-4ebc-94e4-686d3357642f-plugins-config-volume\") pod \"80b81ee0-c86f-4ebc-94e4-686d3357642f\" (UID: \"80b81ee0-c86f-4ebc-94e4-686d3357642f\") "
Nov 03 10:40:41.219491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.219374    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b81ee0-c86f-4ebc-94e4-686d3357642f-kube-api-access-9p7kp\" (UniqueName: \"kubernetes.io/projected/80b81ee0-c86f-4ebc-94e4-686d3357642f-kube-api-access-9p7kp\") pod \"80b81ee0-c86f-4ebc-94e4-686d3357642f\" (UID: \"80b81ee0-c86f-4ebc-94e4-686d3357642f\") "
Nov 03 10:40:41.219815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.219789    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44d293f-a431-4bd0-a278-29a79593a981-plugins-config-volume" pod "b44d293f-a431-4bd0-a278-29a79593a981" (UID: "b44d293f-a431-4bd0-a278-29a79593a981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:41.220147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.220125    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b81ee0-c86f-4ebc-94e4-686d3357642f-plugins-config-volume" pod "80b81ee0-c86f-4ebc-94e4-686d3357642f" (UID: "80b81ee0-c86f-4ebc-94e4-686d3357642f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:41.222103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.222035    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44d293f-a431-4bd0-a278-29a79593a981-kube-api-access-7s5tz" pod "b44d293f-a431-4bd0-a278-29a79593a981" (UID: "b44d293f-a431-4bd0-a278-29a79593a981"). InnerVolumeSpecName "kube-api-access-7s5tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:41.222235 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.222215    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b81ee0-c86f-4ebc-94e4-686d3357642f-kube-api-access-9p7kp" pod "80b81ee0-c86f-4ebc-94e4-686d3357642f" (UID: "80b81ee0-c86f-4ebc-94e4-686d3357642f"). InnerVolumeSpecName "kube-api-access-9p7kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:41.320292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.320262    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44d293f-a431-4bd0-a278-29a79593a981-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:41.320292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.320286    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b81ee0-c86f-4ebc-94e4-686d3357642f-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:41.320292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.320296    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p7kp\" (UniqueName: \"kubernetes.io/projected/80b81ee0-c86f-4ebc-94e4-686d3357642f-kube-api-access-9p7kp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:41.320496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.320306    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s5tz\" (UniqueName: \"kubernetes.io/projected/b44d293f-a431-4bd0-a278-29a79593a981-kube-api-access-7s5tz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:41.374095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.374058    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:41.454183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.454149    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"]
Nov 03 10:40:41.547303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.547273    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:41.619198 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.619158    5817 generic.go:356] "Generic (PLEG): container finished" podID="80b81ee0-c86f-4ebc-94e4-686d3357642f" containerID="f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8" exitCode=0
Nov 03 10:40:41.619386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.619225    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"
Nov 03 10:40:41.619386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.619239    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8" event={"ID":"80b81ee0-c86f-4ebc-94e4-686d3357642f","Type":"ContainerDied","Data":"f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8"}
Nov 03 10:40:41.619386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.619292    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8" event={"ID":"80b81ee0-c86f-4ebc-94e4-686d3357642f","Type":"ContainerDied","Data":"4a808c70be9ca82cf31d8d64fc422bf9676884f1b8c0feb2a71ed6e5e4bad7af"}
Nov 03 10:40:41.619386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.619315    5817 scope.go:122] "RemoveContainer" containerID="f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8"
Nov 03 10:40:41.621003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.620963    5817 generic.go:356] "Generic (PLEG): container finished" podID="b44d293f-a431-4bd0-a278-29a79593a981" containerID="49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5" exitCode=0
Nov 03 10:40:41.621124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.621047    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn" event={"ID":"b44d293f-a431-4bd0-a278-29a79593a981","Type":"ContainerDied","Data":"49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5"}
Nov 03 10:40:41.621124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.621094    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn" event={"ID":"b44d293f-a431-4bd0-a278-29a79593a981","Type":"ContainerDied","Data":"64c130c37f90bba5bd78bd8e9f969dcb34e9f60743d7d36aa54b5cf7b8f31f45"}
Nov 03 10:40:41.621124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.621052    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"
Nov 03 10:40:41.636822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.633879    5817 scope.go:122] "RemoveContainer" containerID="f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8"
Nov 03 10:40:41.636822 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:41.636420    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8\": container with ID starting with f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8 not found: ID does not exist" containerID="f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8"
Nov 03 10:40:41.636822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.636449    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8"} err="failed to get container status \"f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8\": rpc error: code = NotFound desc = could not find container \"f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8\": container with ID starting with f63e7d6978b36dc4e0aaef08cbcb1714aa0c1387041728ba83bceea7718f14d8 not found: ID does not exist"
Nov 03 10:40:41.636822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.636464    5817 scope.go:122] "RemoveContainer" containerID="49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5"
Nov 03 10:40:41.646098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.646060    5817 scope.go:122] "RemoveContainer" containerID="49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5"
Nov 03 10:40:41.646349 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:41.646332    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5\": container with ID starting with 49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5 not found: ID does not exist" containerID="49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5"
Nov 03 10:40:41.646390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.646355    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5"} err="failed to get container status \"49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5\": rpc error: code = NotFound desc = could not find container \"49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5\": container with ID starting with 49aeff5a8f3bae44b9d8f2ebd8ea9ddddce2567286388baaa990d0599b789af5 not found: ID does not exist"
Nov 03 10:40:41.650219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.650196    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"]
Nov 03 10:40:41.653854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.653835    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fc81ef02-7c6d4fcbcf-q6vmn"]
Nov 03 10:40:41.663780 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.663762    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"]
Nov 03 10:40:41.670672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.670655    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fc81ef02-7c56555d69-gwxd8"]
Nov 03 10:40:41.696631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.695099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:41.846148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.846076    5817 patch_prober.go:29] interesting pod/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:40:41.846287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.846143    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:40:41.856563 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.856501    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:41.886083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886047    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"]
Nov 03 10:40:41.886527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886513    5817 cpu_manager.go:414] "containerMap: removing container" podUID="46eaf866-9a83-4df3-950e-97299af275b5" containerName="payload-pre-processing"
Nov 03 10:40:41.886607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886528    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46eaf866-9a83-4df3-950e-97299af275b5" containerName="payload-pre-processing"
Nov 03 10:40:41.886607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886544    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b44d293f-a431-4bd0-a278-29a79593a981" containerName="payload-pre-processing"
Nov 03 10:40:41.886607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886550    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44d293f-a431-4bd0-a278-29a79593a981" containerName="payload-pre-processing"
Nov 03 10:40:41.886607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886563    5817 cpu_manager.go:414] "containerMap: removing container" podUID="80b81ee0-c86f-4ebc-94e4-686d3357642f" containerName="payload-processing"
Nov 03 10:40:41.886607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886593    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b81ee0-c86f-4ebc-94e4-686d3357642f" containerName="payload-processing"
Nov 03 10:40:41.886868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886713    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="46eaf866-9a83-4df3-950e-97299af275b5" containerName="payload-pre-processing"
Nov 03 10:40:41.886868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886725    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44d293f-a431-4bd0-a278-29a79593a981" containerName="payload-pre-processing"
Nov 03 10:40:41.886868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.886738    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b81ee0-c86f-4ebc-94e4-686d3357642f" containerName="payload-processing"
Nov 03 10:40:41.892159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.892079    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:41.902418 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.902088    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-010232-openshift-default-dockercfg-txh8q\""
Nov 03 10:40:41.907457 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.907427    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46eaf866-9a83-4df3-950e-97299af275b5" path="/var/lib/kubelet/pods/46eaf866-9a83-4df3-950e-97299af275b5/volumes"
Nov 03 10:40:41.908247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.908224    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b81ee0-c86f-4ebc-94e4-686d3357642f" path="/var/lib/kubelet/pods/80b81ee0-c86f-4ebc-94e4-686d3357642f/volumes"
Nov 03 10:40:41.908694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.908666    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44d293f-a431-4bd0-a278-29a79593a981" path="/var/lib/kubelet/pods/b44d293f-a431-4bd0-a278-29a79593a981/volumes"
Nov 03 10:40:41.909009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.908993    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"]
Nov 03 10:40:41.910800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:41.910780    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:40:42.004639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.004600    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:42.027743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.027704    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.027868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.027768    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.027995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.027971    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.028102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.028011    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd85m\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-kube-api-access-cd85m\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.028102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.028057    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.028263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.028246    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.028326 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.028299    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.028427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.028406    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.028470 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.028447    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129039    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129097    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129134    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129157    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd85m\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-kube-api-access-cd85m\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129189    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129482    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129511    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129594    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129618    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129631    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129674    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129925    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.129997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.129908    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.130060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.130041    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.131642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.131614    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.132478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.132458    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.143847 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.143820    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd85m\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-kube-api-access-cd85m\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.153588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.153540    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.212452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.212427    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:42.241508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.241463    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:42.316392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.316364    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"]
Nov 03 10:40:42.320149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.320120    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-afe85416-wr9l5"]
Nov 03 10:40:42.385128 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.385053    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:42.498607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.498561    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"]
Nov 03 10:40:42.500673 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:42.500646    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bfa55c_c68a_4b70_a943_8bbd93b6fda2.slice/crio-4cdb218c7bb3f2c7f5bb2faf32ece2ed91e2a5fce8968e78ee8022b50b884475 WatchSource:0}: Error finding container 4cdb218c7bb3f2c7f5bb2faf32ece2ed91e2a5fce8968e78ee8022b50b884475: Status 404 returned error can't find the container with id 4cdb218c7bb3f2c7f5bb2faf32ece2ed91e2a5fce8968e78ee8022b50b884475
Nov 03 10:40:42.503117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.503087    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:42.503207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.503150    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:42.503207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.503179    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:42.520634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.520604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:42.635182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.635097    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" event={"ID":"69bfa55c-c68a-4b70-a943-8bbd93b6fda2","Type":"ContainerStarted","Data":"d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7"}
Nov 03 10:40:42.635182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.635159    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" event={"ID":"69bfa55c-c68a-4b70-a943-8bbd93b6fda2","Type":"ContainerStarted","Data":"4cdb218c7bb3f2c7f5bb2faf32ece2ed91e2a5fce8968e78ee8022b50b884475"}
Nov 03 10:40:42.658971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.658941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:42.665161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:42.665116    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podStartSLOduration=1.665101783 podStartE2EDuration="1.665101783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:42.659833802 +0000 UTC m=+672.865415857" watchObservedRunningTime="2026-11-03 10:40:42.665101783 +0000 UTC m=+672.870683904"
Nov 03 10:40:43.028564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.028521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:43.159620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.159563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:43.216300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.212919    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:43.249940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.249909    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:43.262042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.262016    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"]
Nov 03 10:40:43.265065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.265046    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19708dae-5ccqd"]
Nov 03 10:40:43.305650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.302151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:43.456616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.455370    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:43.622650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.619041    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:43.642437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.642411    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:43.643362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.643332    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:40:43.893505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.893421    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:43.920810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.920777    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c474493-16b5-4db4-ae2e-cae433ba058f" path="/var/lib/kubelet/pods/6c474493-16b5-4db4-ae2e-cae433ba058f/volumes"
Nov 03 10:40:43.921259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:43.921239    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69f26cf-f1a9-4959-9d12-d82f71f40658" path="/var/lib/kubelet/pods/c69f26cf-f1a9-4959-9d12-d82f71f40658/volumes"
Nov 03 10:40:44.385754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.385719    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:44.533884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.533848    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:44.663396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.663316    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:44.764926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.764896    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"]
Nov 03 10:40:44.769231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.769210    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.773176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.773157    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e78c33-openshift-default-dockercfg-brqpb\""
Nov 03 10:40:44.783300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.783274    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"]
Nov 03 10:40:44.869630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.869566    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.869779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.869649    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slx65\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-kube-api-access-slx65\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.869779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.869715    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.869779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.869753    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.869779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.869773    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.869969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.869918    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.869969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.869961    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.870063 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.870049    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.870097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.870075    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.912598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.911634    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:44.971210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971171    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slx65\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-kube-api-access-slx65\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.971390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971223    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.971390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971268    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.971390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971291    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.971390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971378    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.971659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971425    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.971659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971471    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.971659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971501    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.971659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971551    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.972031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.971985    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.973756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.973733    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.974226 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.974205    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.974456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.974434    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:44.978591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:44.974951    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:45.010591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.005371    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:45.010591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.006403    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slx65\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-kube-api-access-slx65\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:45.032595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.022056    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:45.034173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.034143    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:45.063367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.063306    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:45.085001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.084977    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:45.311654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.310498    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:45.388637 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.387319    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"]
Nov 03 10:40:45.406077 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:45.406052    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd4b11e_6392_4182_9b06_9a46cce69f7c.slice/crio-5d9f0d5cb4ba666888b2c6922f7fc0f131c572639371dba4c439923307d5fe54 WatchSource:0}: Error finding container 5d9f0d5cb4ba666888b2c6922f7fc0f131c572639371dba4c439923307d5fe54: Status 404 returned error can't find the container with id 5d9f0d5cb4ba666888b2c6922f7fc0f131c572639371dba4c439923307d5fe54
Nov 03 10:40:45.414282 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.414244    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:45.414381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.414328    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:45.414381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.414375    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:45.506402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.505636    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr" podUID="f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c" containerName="payload-processing" containerID="cri-o://6b57a83714625f9e39d289be55fdbfce3ee92a0ca7a80b010292cc8c3f91b28c" gracePeriod=25
Nov 03 10:40:45.518656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.509808    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6" podUID="9235dbab-c8e3-4786-8789-46f194cd9937" containerName="payload-pre-processing" containerID="cri-o://bb353a674181b91c75b2e44a4d86e056cd922232fbe6e732f6f310cda591729c" gracePeriod=25
Nov 03 10:40:45.631751 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.631505    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:45.658617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.658234    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" event={"ID":"dbd4b11e-6392-4182-9b06-9a46cce69f7c","Type":"ContainerStarted","Data":"a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09"}
Nov 03 10:40:45.658617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.658288    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" event={"ID":"dbd4b11e-6392-4182-9b06-9a46cce69f7c","Type":"ContainerStarted","Data":"5d9f0d5cb4ba666888b2c6922f7fc0f131c572639371dba4c439923307d5fe54"}
Nov 03 10:40:45.661188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.660659    5817 generic.go:356] "Generic (PLEG): container finished" podID="f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c" containerID="6b57a83714625f9e39d289be55fdbfce3ee92a0ca7a80b010292cc8c3f91b28c" exitCode=0
Nov 03 10:40:45.661188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.660799    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr" event={"ID":"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c","Type":"ContainerDied","Data":"6b57a83714625f9e39d289be55fdbfce3ee92a0ca7a80b010292cc8c3f91b28c"}
Nov 03 10:40:45.664398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.664372    5817 generic.go:356] "Generic (PLEG): container finished" podID="9235dbab-c8e3-4786-8789-46f194cd9937" containerID="bb353a674181b91c75b2e44a4d86e056cd922232fbe6e732f6f310cda591729c" exitCode=0
Nov 03 10:40:45.665418 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.665392    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6" event={"ID":"9235dbab-c8e3-4786-8789-46f194cd9937","Type":"ContainerDied","Data":"bb353a674181b91c75b2e44a4d86e056cd922232fbe6e732f6f310cda591729c"}
Nov 03 10:40:45.692099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.684608    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podStartSLOduration=1.684591363 podStartE2EDuration="1.684591363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:45.68291903 +0000 UTC m=+675.888501081" watchObservedRunningTime="2026-11-03 10:40:45.684591363 +0000 UTC m=+675.890173415"
Nov 03 10:40:45.784178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.784117    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:45.953085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:45.952833    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:46.085450 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.085347    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:46.117279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.115980    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:46.140011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.139983    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:46.146716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.146695    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:46.152146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.152090    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:46.229177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.229039    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:46.330682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.330638    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-podinfo\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.330845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.330697    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-certs\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.330845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.330736    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-plugins-config-volume\") pod \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\" (UID: \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\") "
Nov 03 10:40:46.330845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.330808    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-socket\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.331023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.330848    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istiod-ca-cert\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.331023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.330877    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-data\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.331023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.330905    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-kube-api-access-hxq6w\" (UniqueName: \"kubernetes.io/projected/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-kube-api-access-hxq6w\") pod \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\" (UID: \"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c\") "
Nov 03 10:40:46.331023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.330941    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9235dbab-c8e3-4786-8789-46f194cd9937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9235dbab-c8e3-4786-8789-46f194cd9937-plugins-config-volume\") pod \"9235dbab-c8e3-4786-8789-46f194cd9937\" (UID: \"9235dbab-c8e3-4786-8789-46f194cd9937\") "
Nov 03 10:40:46.331023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.331000    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-kube-api-access-rrq98\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-kube-api-access-rrq98\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.331260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.331042    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-envoy\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.331260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.331096    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-credential-socket\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.331260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.331161    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9235dbab-c8e3-4786-8789-46f194cd9937-kube-api-access-qbxfr\" (UniqueName: \"kubernetes.io/projected/9235dbab-c8e3-4786-8789-46f194cd9937-kube-api-access-qbxfr\") pod \"9235dbab-c8e3-4786-8789-46f194cd9937\" (UID: \"9235dbab-c8e3-4786-8789-46f194cd9937\") "
Nov 03 10:40:46.331260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.331190    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-token\") pod \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\" (UID: \"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8\") "
Nov 03 10:40:46.331442 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.331377    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-certs" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.331966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.331561    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.331966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.331817    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-data" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.333289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.332647    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istiod-ca-cert" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:46.333289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.332748    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9235dbab-c8e3-4786-8789-46f194cd9937-plugins-config-volume" pod "9235dbab-c8e3-4786-8789-46f194cd9937" (UID: "9235dbab-c8e3-4786-8789-46f194cd9937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:46.333289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.332925    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-plugins-config-volume" pod "f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c" (UID: "f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:46.333289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.333017    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-socket" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.333289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.333196    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-credential-socket" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.336346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.336304    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-podinfo" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:40:46.336524 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.336513    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-envoy" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.337200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.337174    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-kube-api-access-hxq6w" pod "f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c" (UID: "f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c"). InnerVolumeSpecName "kube-api-access-hxq6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:46.337639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.337548    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9235dbab-c8e3-4786-8789-46f194cd9937-kube-api-access-qbxfr" pod "9235dbab-c8e3-4786-8789-46f194cd9937" (UID: "9235dbab-c8e3-4786-8789-46f194cd9937"). InnerVolumeSpecName "kube-api-access-qbxfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:46.337639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.337090    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-token" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:46.337986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.337825    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-kube-api-access-rrq98" pod "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" (UID: "20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8"). InnerVolumeSpecName "kube-api-access-rrq98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:46.348008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.347746    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:46.375556 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.375532    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:46.438599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438212    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-certs\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.438599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438272    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fec804-7104-45c8-8c9b-9a901c1505eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fec804-7104-45c8-8c9b-9a901c1505eb-istiod-ca-cert\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.438599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438315    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-kube-api-access-5jz79\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-kube-api-access-5jz79\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.438599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438356    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-podinfo\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.438599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438412    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-token\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-token\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.438599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438457    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-credential-socket\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.438599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438494    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-data\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438640    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-envoy\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.438713    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-socket\") pod \"a0fec804-7104-45c8-8c9b-9a901c1505eb\" (UID: \"a0fec804-7104-45c8-8c9b-9a901c1505eb\") "
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439060    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439079    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439095    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439109    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxq6w\" (UniqueName: \"kubernetes.io/projected/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-kube-api-access-hxq6w\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439125    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9235dbab-c8e3-4786-8789-46f194cd9937-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439139    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrq98\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-kube-api-access-rrq98\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439153    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439166    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439180    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbxfr\" (UniqueName: \"kubernetes.io/projected/9235dbab-c8e3-4786-8789-46f194cd9937-kube-api-access-qbxfr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439194    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439207    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.439220    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.449067    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-socket" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.452100    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fec804-7104-45c8-8c9b-9a901c1505eb-istiod-ca-cert" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.452361    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-certs" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.452858    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-credential-socket" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.454804    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-data" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.454885    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85" podUID="53eba721-0edd-49ad-ad28-7f9d874d4839" containerName="payload-pre-processing" containerID="cri-o://4ee0f40bffdc30309633becc4eb94e28f1e8f78b5bbf9dfc30b730f59b08c98b" gracePeriod=25
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.457156    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-token" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.460494    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-podinfo" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.460983    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-envoy" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:40:46.464584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.461908    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-kube-api-access-5jz79" pod "a0fec804-7104-45c8-8c9b-9a901c1505eb" (UID: "a0fec804-7104-45c8-8c9b-9a901c1505eb"). InnerVolumeSpecName "kube-api-access-5jz79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:46.539927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539841    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.539927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539866    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.539927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539875    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fec804-7104-45c8-8c9b-9a901c1505eb-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.539927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539884    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jz79\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-kube-api-access-5jz79\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.539927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539895    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.539927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539903    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.539927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539930    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.540392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539939    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.540392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.539947    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fec804-7104-45c8-8c9b-9a901c1505eb-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:46.576236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.576207    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:40:46.658672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.658633    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:46.679680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.679644    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr" event={"ID":"f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c","Type":"ContainerDied","Data":"4112f97a305e530705786151586c024275458d1d08bace836062cb58096ec3a1"}
Nov 03 10:40:46.711986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.711941    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6" event={"ID":"9235dbab-c8e3-4786-8789-46f194cd9937","Type":"ContainerDied","Data":"7796f7b188fa53d428cebf04b307756ffafb6b229fce37f1fcc01d4f62630eb4"}
Nov 03 10:40:46.712221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.696636    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"
Nov 03 10:40:46.714438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.714416    5817 scope.go:122] "RemoveContainer" containerID="6b57a83714625f9e39d289be55fdbfce3ee92a0ca7a80b010292cc8c3f91b28c"
Nov 03 10:40:46.714707 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.679710    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"
Nov 03 10:40:46.750125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.750095    5817 generic.go:356] "Generic (PLEG): container finished" podID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerID="4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653" exitCode=0
Nov 03 10:40:46.750230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.750197    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" event={"ID":"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8","Type":"ContainerDied","Data":"4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653"}
Nov 03 10:40:46.750615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.750245    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx" event={"ID":"20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8","Type":"ContainerDied","Data":"326b64dd91fa5b58b1f92566cc3870c877af0444e0041745afee7dafacb26ea0"}
Nov 03 10:40:46.750615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.750258    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"
Nov 03 10:40:46.761679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.761616    5817 generic.go:356] "Generic (PLEG): container finished" podID="53eba721-0edd-49ad-ad28-7f9d874d4839" containerID="4ee0f40bffdc30309633becc4eb94e28f1e8f78b5bbf9dfc30b730f59b08c98b" exitCode=0
Nov 03 10:40:46.761817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.761793    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85" event={"ID":"53eba721-0edd-49ad-ad28-7f9d874d4839","Type":"ContainerDied","Data":"4ee0f40bffdc30309633becc4eb94e28f1e8f78b5bbf9dfc30b730f59b08c98b"}
Nov 03 10:40:46.770526 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.770495    5817 generic.go:356] "Generic (PLEG): container finished" podID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerID="754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8" exitCode=0
Nov 03 10:40:46.773620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.772220    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"
Nov 03 10:40:46.773620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.772857    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" event={"ID":"a0fec804-7104-45c8-8c9b-9a901c1505eb","Type":"ContainerDied","Data":"754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8"}
Nov 03 10:40:46.773620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.772911    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:46.773620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.772937    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8" event={"ID":"a0fec804-7104-45c8-8c9b-9a901c1505eb","Type":"ContainerDied","Data":"e50184f0f3d7b226ace91a3bb1444b690e60dc6cd4df1ac3fc29d659f7af8695"}
Nov 03 10:40:46.774823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.774451    5817 scope.go:122] "RemoveContainer" containerID="bb353a674181b91c75b2e44a4d86e056cd922232fbe6e732f6f310cda591729c"
Nov 03 10:40:46.774823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.774528    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"]
Nov 03 10:40:46.774823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.774632    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:40:46.777479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.777457    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19708dae-85fbf668cf-jthf6"]
Nov 03 10:40:46.784177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.784160    5817 scope.go:122] "RemoveContainer" containerID="4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653"
Nov 03 10:40:46.819911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.819618    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"]
Nov 03 10:40:46.828915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.828877    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fc81ef02-openshift-default-69d9fc6bb9-57zwx"]
Nov 03 10:40:46.830450 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.830422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:46.835179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.835163    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:46.836339 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.836114    5817 scope.go:122] "RemoveContainer" containerID="558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826"
Nov 03 10:40:46.837520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.837501    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"]
Nov 03 10:40:46.840706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.840687    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19708dae-b9dd77bfc-988nr"]
Nov 03 10:40:46.847362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.847339    5817 scope.go:122] "RemoveContainer" containerID="4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653"
Nov 03 10:40:46.847732 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:46.847713    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653\": container with ID starting with 4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653 not found: ID does not exist" containerID="4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653"
Nov 03 10:40:46.847816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.847753    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653"} err="failed to get container status \"4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653\": rpc error: code = NotFound desc = could not find container \"4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653\": container with ID starting with 4d9bf428d9643849d82c082092cb29148239ad7b8f12d874ec828f7909488653 not found: ID does not exist"
Nov 03 10:40:46.847816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.847771    5817 scope.go:122] "RemoveContainer" containerID="558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826"
Nov 03 10:40:46.848090 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:46.848071    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826\": container with ID starting with 558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826 not found: ID does not exist" containerID="558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826"
Nov 03 10:40:46.848153 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.848096    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826"} err="failed to get container status \"558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826\": rpc error: code = NotFound desc = could not find container \"558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826\": container with ID starting with 558220ba93ef9af0b5b25685885a9379bf7aade674195670d11c9e82f5185826 not found: ID does not exist"
Nov 03 10:40:46.848153 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.848113    5817 scope.go:122] "RemoveContainer" containerID="754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8"
Nov 03 10:40:46.858101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.858078    5817 scope.go:122] "RemoveContainer" containerID="3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242"
Nov 03 10:40:46.868444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.868386    5817 scope.go:122] "RemoveContainer" containerID="754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8"
Nov 03 10:40:46.868694 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:46.868646    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8\": container with ID starting with 754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8 not found: ID does not exist" containerID="754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8"
Nov 03 10:40:46.868694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.868680    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8"} err="failed to get container status \"754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8\": rpc error: code = NotFound desc = could not find container \"754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8\": container with ID starting with 754f41e59097d47da9b9aa549a1343346f4e1ebf49f65663d85966dbaae1cdf8 not found: ID does not exist"
Nov 03 10:40:46.868885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.868704    5817 scope.go:122] "RemoveContainer" containerID="3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242"
Nov 03 10:40:46.868977 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:46.868946    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242\": container with ID starting with 3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242 not found: ID does not exist" containerID="3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242"
Nov 03 10:40:46.868977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.868964    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242"} err="failed to get container status \"3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242\": rpc error: code = NotFound desc = could not find container \"3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242\": container with ID starting with 3fb5694ba951d5c3cae48de516596656efa764072b6efdc7d8c9cfbfd1715242 not found: ID does not exist"
Nov 03 10:40:46.876463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.876404    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"]
Nov 03 10:40:46.880825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.880804    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3a6822cf-openshift-default-5cdf876bc9-xkfk8"]
Nov 03 10:40:46.945446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.945419    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53eba721-0edd-49ad-ad28-7f9d874d4839-kube-api-access-9zv6w\" (UniqueName: \"kubernetes.io/projected/53eba721-0edd-49ad-ad28-7f9d874d4839-kube-api-access-9zv6w\") pod \"53eba721-0edd-49ad-ad28-7f9d874d4839\" (UID: \"53eba721-0edd-49ad-ad28-7f9d874d4839\") "
Nov 03 10:40:46.945638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.945625    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53eba721-0edd-49ad-ad28-7f9d874d4839-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53eba721-0edd-49ad-ad28-7f9d874d4839-plugins-config-volume\") pod \"53eba721-0edd-49ad-ad28-7f9d874d4839\" (UID: \"53eba721-0edd-49ad-ad28-7f9d874d4839\") "
Nov 03 10:40:46.946263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.946243    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53eba721-0edd-49ad-ad28-7f9d874d4839-plugins-config-volume" pod "53eba721-0edd-49ad-ad28-7f9d874d4839" (UID: "53eba721-0edd-49ad-ad28-7f9d874d4839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:46.947860 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.947835    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53eba721-0edd-49ad-ad28-7f9d874d4839-kube-api-access-9zv6w" pod "53eba721-0edd-49ad-ad28-7f9d874d4839" (UID: "53eba721-0edd-49ad-ad28-7f9d874d4839"). InnerVolumeSpecName "kube-api-access-9zv6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:46.971191 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:46.971161    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:47.051944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.050865    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zv6w\" (UniqueName: \"kubernetes.io/projected/53eba721-0edd-49ad-ad28-7f9d874d4839-kube-api-access-9zv6w\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:47.051944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.050900    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53eba721-0edd-49ad-ad28-7f9d874d4839-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.128649    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"]
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129302    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129321    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129334    5817 cpu_manager.go:414] "containerMap: removing container" podUID="53eba721-0edd-49ad-ad28-7f9d874d4839" containerName="payload-pre-processing"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129342    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53eba721-0edd-49ad-ad28-7f9d874d4839" containerName="payload-pre-processing"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129359    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129366    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129381    5817 cpu_manager.go:414] "containerMap: removing container" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129389    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129403    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c" containerName="payload-processing"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129411    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c" containerName="payload-processing"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129458    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9235dbab-c8e3-4786-8789-46f194cd9937" containerName="payload-pre-processing"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129465    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9235dbab-c8e3-4786-8789-46f194cd9937" containerName="payload-pre-processing"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129559    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9235dbab-c8e3-4786-8789-46f194cd9937" containerName="payload-pre-processing"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129592    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129604    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129613    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="53eba721-0edd-49ad-ad28-7f9d874d4839" containerName="payload-pre-processing"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129623    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" containerName="istio-proxy"
Nov 03 10:40:47.131386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.129633    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c" containerName="payload-processing"
Nov 03 10:40:47.136426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.136273    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:47.136800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.136690    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:47.138728 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.136554    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"]
Nov 03 10:40:47.142021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.142002    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d5d3c3\""
Nov 03 10:40:47.152885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.152834    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g\" (UID: \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:47.152885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.152868    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9m79\" (UniqueName: \"kubernetes.io/projected/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-kube-api-access-x9m79\") pod \"payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g\" (UID: \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:47.258593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.258126    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g\" (UID: \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:47.258593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.258193    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9m79\" (UniqueName: \"kubernetes.io/projected/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-kube-api-access-x9m79\") pod \"payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g\" (UID: \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:47.259504 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.259451    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g\" (UID: \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:47.270972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.270907    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9m79\" (UniqueName: \"kubernetes.io/projected/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-kube-api-access-x9m79\") pod \"payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g\" (UID: \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:47.277859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.277823    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:47.453248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.453221    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:47.642736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.642705    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 10:40:47.730915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.727651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:47.764900 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.764875    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"]
Nov 03 10:40:47.766402 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:47.766380    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f128c57_8e1f_4ee4_8ab3_2b031ba7d388.slice/crio-3d87e2610dcf2086d355a1d6996f89fe95f66180af31290c33fd2b714db7e7c2 WatchSource:0}: Error finding container 3d87e2610dcf2086d355a1d6996f89fe95f66180af31290c33fd2b714db7e7c2: Status 404 returned error can't find the container with id 3d87e2610dcf2086d355a1d6996f89fe95f66180af31290c33fd2b714db7e7c2
Nov 03 10:40:47.825594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.825494    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"
Nov 03 10:40:47.825594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.825518    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85" event={"ID":"53eba721-0edd-49ad-ad28-7f9d874d4839","Type":"ContainerDied","Data":"e91f6ebb35851faf19f0e7911172663b7dda8991942e12483138c11c7dec778f"}
Nov 03 10:40:47.825952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.825907    5817 scope.go:122] "RemoveContainer" containerID="4ee0f40bffdc30309633becc4eb94e28f1e8f78b5bbf9dfc30b730f59b08c98b"
Nov 03 10:40:47.828874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.828850    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" event={"ID":"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388","Type":"ContainerStarted","Data":"3d87e2610dcf2086d355a1d6996f89fe95f66180af31290c33fd2b714db7e7c2"}
Nov 03 10:40:47.866796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.866635    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"]
Nov 03 10:40:47.882719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.882693    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-afe85416-6df9446b99-kzw85"]
Nov 03 10:40:47.923593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.923551    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" path="/var/lib/kubelet/pods/20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8/volumes"
Nov 03 10:40:47.924138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.924122    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53eba721-0edd-49ad-ad28-7f9d874d4839" path="/var/lib/kubelet/pods/53eba721-0edd-49ad-ad28-7f9d874d4839/volumes"
Nov 03 10:40:47.924554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.924541    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9235dbab-c8e3-4786-8789-46f194cd9937" path="/var/lib/kubelet/pods/9235dbab-c8e3-4786-8789-46f194cd9937/volumes"
Nov 03 10:40:47.925340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.925325    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fec804-7104-45c8-8c9b-9a901c1505eb" path="/var/lib/kubelet/pods/a0fec804-7104-45c8-8c9b-9a901c1505eb/volumes"
Nov 03 10:40:47.925782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:47.925769    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c" path="/var/lib/kubelet/pods/f63a2e32-1ea0-42bb-96ca-8f4fbe44f96c/volumes"
Nov 03 10:40:48.135394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:48.135300    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:48.337549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:48.337504    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:48.650109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:48.650071    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:48.796114 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:48.796082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:48.836550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:48.836518    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" event={"ID":"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388","Type":"ContainerStarted","Data":"135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32"}
Nov 03 10:40:48.836848 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:48.836775    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:48.854444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:48.854402    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" podStartSLOduration=1.854379268 podStartE2EDuration="1.854379268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:48.852813182 +0000 UTC m=+679.058395242" watchObservedRunningTime="2026-11-03 10:40:48.854379268 +0000 UTC m=+679.059961320"
Nov 03 10:40:49.340706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:49.340668    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:49.482993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:49.482917    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:49.594281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:49.594201    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 10:40:49.594281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:49.594271    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 10:40:49.743162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:49.743037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:50.365261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:50.365224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:50.586763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:50.586710    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:40:50.586950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:50.586772    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:40:51.161396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:51.161369    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:51.288173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:51.288128    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:51.434343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:51.434317    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:52.082731 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:52.082623    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:52.478861 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:40:52.478824    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:40:52.820656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:52.818794    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:52.868342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:52.868309    5817 generic.go:356] "Generic (PLEG): container finished" podID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerID="adf053cb278ded75802f7c25edd396ee1024a08f3c1991b88479071bbea0d83c" exitCode=0
Nov 03 10:40:52.868503 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:52.868386    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" event={"ID":"235f4586-fb1b-400d-9a25-fb73a5aa5630","Type":"ContainerDied","Data":"adf053cb278ded75802f7c25edd396ee1024a08f3c1991b88479071bbea0d83c"}
Nov 03 10:40:52.868879 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:52.868860    5817 scope.go:122] "RemoveContainer" containerID="adf053cb278ded75802f7c25edd396ee1024a08f3c1991b88479071bbea0d83c"
Nov 03 10:40:52.870267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:52.870243    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:52.870327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:52.870307    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:52.870369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:52.870335    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:53.585350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:53.585316    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:53.876611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:53.876518    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" event={"ID":"235f4586-fb1b-400d-9a25-fb73a5aa5630","Type":"ContainerStarted","Data":"22335baf83eef623ce60c4332cf236a8bd188af309cd0de381dc0418b7600892"}
Nov 03 10:40:53.978124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:53.978094    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:53.978316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:53.978294    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 10:40:53.978368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:53.978341    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 10:40:54.292030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:54.291996    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:54.315594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:54.315547    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:40:54.980594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:54.979999    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 10:40:54.980594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:54.980084    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 10:40:55.175519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:55.175486    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:55.978756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:55.978724    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 10:40:55.979050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:55.979015    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 10:40:56.099758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:56.099722    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:56.925599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:56.925380    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:56.979780 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:56.979745    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 10:40:56.980095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:56.980049    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 10:40:57.757759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:57.757655    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:57.950517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:57.950484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:57.983617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:57.981108    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 10:40:57.983617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:57.982413    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 10:40:58.775266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.775237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:58.962089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.960422    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"]
Nov 03 10:40:58.962089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.960926    5817 cpu_manager.go:414] "containerMap: removing container" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy"
Nov 03 10:40:58.962089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.960939    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy"
Nov 03 10:40:58.962089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.961024    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="20561cb0-7e1d-4e2a-a8de-ba7aa6202ee8" containerName="istio-proxy"
Nov 03 10:40:58.966274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.966253    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:58.974187 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.973668    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f78621\""
Nov 03 10:40:58.974930 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.974907    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"]
Nov 03 10:40:58.978559 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.978529    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 10:40:58.978559 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:58.978601    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 10:40:59.035672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.033849    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:59.039937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.039894    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"]
Nov 03 10:40:59.050597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.048125    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"]
Nov 03 10:40:59.050597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.048249    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.052141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.052104    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f78621-dockercfg-bz8kw\""
Nov 03 10:40:59.072003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.071934    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbvv\" (UniqueName: \"kubernetes.io/projected/2d28be58-89ea-4734-a9a7-f11bbd106db2-kube-api-access-xbbvv\") pod \"payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt\" (UID: \"2d28be58-89ea-4734-a9a7-f11bbd106db2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:59.072273 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.072168    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d28be58-89ea-4734-a9a7-f11bbd106db2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt\" (UID: \"2d28be58-89ea-4734-a9a7-f11bbd106db2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:59.072405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.072337    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt\" (UID: \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.072468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.072442    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbrr\" (UniqueName: \"kubernetes.io/projected/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-kube-api-access-vkbrr\") pod \"payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt\" (UID: \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.166172 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.166146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:59.174475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.173471    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt\" (UID: \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.174475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.173508    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbrr\" (UniqueName: \"kubernetes.io/projected/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-kube-api-access-vkbrr\") pod \"payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt\" (UID: \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.174475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.173588    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbvv\" (UniqueName: \"kubernetes.io/projected/2d28be58-89ea-4734-a9a7-f11bbd106db2-kube-api-access-xbbvv\") pod \"payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt\" (UID: \"2d28be58-89ea-4734-a9a7-f11bbd106db2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:59.174475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.173616    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d28be58-89ea-4734-a9a7-f11bbd106db2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt\" (UID: \"2d28be58-89ea-4734-a9a7-f11bbd106db2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:59.174475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.174274    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d28be58-89ea-4734-a9a7-f11bbd106db2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt\" (UID: \"2d28be58-89ea-4734-a9a7-f11bbd106db2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:59.174781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.174627    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt\" (UID: \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.182722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.182687    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbrr\" (UniqueName: \"kubernetes.io/projected/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-kube-api-access-vkbrr\") pod \"payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt\" (UID: \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.183448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.183425    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbvv\" (UniqueName: \"kubernetes.io/projected/2d28be58-89ea-4734-a9a7-f11bbd106db2-kube-api-access-xbbvv\") pod \"payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt\" (UID: \"2d28be58-89ea-4734-a9a7-f11bbd106db2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:59.285834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.285738    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:59.372423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.372386    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.559145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.556907    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:40:59.654379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.654331    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"]
Nov 03 10:40:59.654662 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:59.654637    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d28be58_89ea_4734_a9a7_f11bbd106db2.slice/crio-5a5e2d7f960210ce15ee8018b8a12088e71578f683e71f44b00e2ab91cc1d294 WatchSource:0}: Error finding container 5a5e2d7f960210ce15ee8018b8a12088e71578f683e71f44b00e2ab91cc1d294: Status 404 returned error can't find the container with id 5a5e2d7f960210ce15ee8018b8a12088e71578f683e71f44b00e2ab91cc1d294
Nov 03 10:40:59.691599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.686769    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:40:59.691599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.686874    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:40:59.759274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.759246    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"]
Nov 03 10:40:59.759643 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:40:59.759618    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2aa5d5b_afba_46ef_b7fc_fa9cf57a2472.slice/crio-9e8724b26ef9041b2679c59994d4cd03d27ad74224cd9f44530f39e0daa2d151 WatchSource:0}: Error finding container 9e8724b26ef9041b2679c59994d4cd03d27ad74224cd9f44530f39e0daa2d151: Status 404 returned error can't find the container with id 9e8724b26ef9041b2679c59994d4cd03d27ad74224cd9f44530f39e0daa2d151
Nov 03 10:40:59.849040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.849010    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 10:40:59.977550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.977499    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt" event={"ID":"2d28be58-89ea-4734-a9a7-f11bbd106db2","Type":"ContainerStarted","Data":"c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3"}
Nov 03 10:40:59.977756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.977664    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt" event={"ID":"2d28be58-89ea-4734-a9a7-f11bbd106db2","Type":"ContainerStarted","Data":"5a5e2d7f960210ce15ee8018b8a12088e71578f683e71f44b00e2ab91cc1d294"}
Nov 03 10:40:59.977756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.977713    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:40:59.980201 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.980167    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt" event={"ID":"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472","Type":"ContainerStarted","Data":"b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15"}
Nov 03 10:40:59.980390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.980227    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt" event={"ID":"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472","Type":"ContainerStarted","Data":"9e8724b26ef9041b2679c59994d4cd03d27ad74224cd9f44530f39e0daa2d151"}
Nov 03 10:40:59.980951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.980930    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:40:59.982967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.982946    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:59.983359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.983339    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:40:59.984313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:40:59.984290    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:41:00.027588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:00.026208    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:00.037435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:00.037314    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt" podStartSLOduration=2.037294536 podStartE2EDuration="2.037294536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:00.01584067 +0000 UTC m=+690.221422725" watchObservedRunningTime="2026-11-03 10:41:00.037294536 +0000 UTC m=+690.242876590"
Nov 03 10:41:00.416472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:00.416430    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:00.644981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:00.644946    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:00.806621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:00.798661    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:00.932116 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:00.932069    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:01.417426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:01.417383    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:01.700892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:01.700302    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:01.848136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:01.848095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:02.609448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:02.609416    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:02.814797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:02.814713    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:41:02.814980 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:02.814798    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:41:03.041223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:03.041184    5817 generic.go:356] "Generic (PLEG): container finished" podID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerID="d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7" exitCode=0
Nov 03 10:41:03.041223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:03.041219    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" event={"ID":"69bfa55c-c68a-4b70-a943-8bbd93b6fda2","Type":"ContainerDied","Data":"d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7"}
Nov 03 10:41:03.041661 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:03.041641    5817 scope.go:122] "RemoveContainer" containerID="d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7"
Nov 03 10:41:03.043091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:03.043058    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:03.043180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:03.043151    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:03.043242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:03.043224    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:03.065124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:03.065075    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt" podStartSLOduration=5.065054395 podStartE2EDuration="5.065054395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:00.06544356 +0000 UTC m=+690.271025613" watchObservedRunningTime="2026-11-03 10:41:03.065054395 +0000 UTC m=+693.270636447"
Nov 03 10:41:03.367227 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:03.367132    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:04.051038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:04.050999    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" event={"ID":"69bfa55c-c68a-4b70-a943-8bbd93b6fda2","Type":"ContainerStarted","Data":"aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b"}
Nov 03 10:41:04.134534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:04.131522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:04.219038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:04.219007    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:41:04.219359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:04.219338    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:41:04.219526 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:04.219502    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 10:41:04.688505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:04.688469    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:41:05.002514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:04.998107    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:05.214597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:05.213433    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:41:05.214597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:05.213489    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 10:41:05.973199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:05.973145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:06.065054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.065026    5817 generic.go:356] "Generic (PLEG): container finished" podID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerID="a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09" exitCode=0
Nov 03 10:41:06.065222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.065074    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" event={"ID":"dbd4b11e-6392-4182-9b06-9a46cce69f7c","Type":"ContainerDied","Data":"a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09"}
Nov 03 10:41:06.065475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.065459    5817 scope.go:122] "RemoveContainer" containerID="a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09"
Nov 03 10:41:06.066782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.066755    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:06.066853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.066820    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:06.066890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.066852    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:06.215597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.215364    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:41:06.215597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.215424    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 10:41:06.851249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.851214    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"]
Nov 03 10:41:06.855089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.855068    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:06.863060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.863034    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"simulator-w4-d5d3c3-kserve-self-signed-certs\""
Nov 03 10:41:06.863163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.863125    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"kube-root-ca.crt\""
Nov 03 10:41:06.863206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.863192    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"default-dockercfg-j8cg6\""
Nov 03 10:41:06.864085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.863893    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"openshift-service-ca.crt\""
Nov 03 10:41:06.867097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.867075    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"]
Nov 03 10:41:06.894598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.887046    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tls-certs\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:06.915599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.894919    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwldd\" (UniqueName: \"kubernetes.io/projected/4fe82e65-7284-4584-9ebe-65b1b6338eaa-kube-api-access-jwldd\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:06.915982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.915949    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-model-cache\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:06.916144 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.916125    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-dshm\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:06.916287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.916273    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tmp-dir\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:06.916420 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.916406    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-home\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:06.942606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:06.942464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:07.022600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.018245    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwldd\" (UniqueName: \"kubernetes.io/projected/4fe82e65-7284-4584-9ebe-65b1b6338eaa-kube-api-access-jwldd\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.032608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.028769    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-model-cache\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.032608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.028818    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-dshm\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.032608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.028886    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tmp-dir\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.032608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.028957    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-home\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.033723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.033689    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tmp-dir\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.048729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.048696    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tls-certs\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.048987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.048960    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-model-cache\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.049096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.048732    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-dshm\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.064864 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.064834    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-home\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.069818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.069790    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tls-certs\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.070385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.070358    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwldd\" (UniqueName: \"kubernetes.io/projected/4fe82e65-7284-4584-9ebe-65b1b6338eaa-kube-api-access-jwldd\") pod \"simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.077977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.077939    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" event={"ID":"dbd4b11e-6392-4182-9b06-9a46cce69f7c","Type":"ContainerStarted","Data":"29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811"}
Nov 03 10:41:07.082117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.082085    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:07.086046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.086009    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:41:07.086312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.086291    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:41:07.101643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.099608    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 10:41:07.169079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.169040    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:07.216493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.216020    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:41:07.216493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.216112    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 10:41:07.510599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.507773    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"]
Nov 03 10:41:07.579464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:07.579433    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:08.085427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.085396    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:41:08.085633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.085444    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 10:41:08.087351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.086745    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" event={"ID":"4fe82e65-7284-4584-9ebe-65b1b6338eaa","Type":"ContainerStarted","Data":"d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12"}
Nov 03 10:41:08.087351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.086794    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" event={"ID":"4fe82e65-7284-4584-9ebe-65b1b6338eaa","Type":"ContainerStarted","Data":"5ac394a14fe8b21dbaa4d94c376b878fafe8e33e7501c744bb4b4d0cd104a3e8"}
Nov 03 10:41:08.170050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.170022    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:08.214934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.214899    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:41:08.215209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.215176    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 10:41:08.279505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.279383    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:08.978140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:08.978110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:09.085690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:09.085654    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:41:09.085896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:09.085731    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 10:41:09.677908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:09.677867    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:09.867407 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:09.867376    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:41:09.867882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:09.867863    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:41:09.869168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:09.869135    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:41:09.918258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:09.918208    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podStartSLOduration=3.9181903240000002 podStartE2EDuration="3.918190324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:08.124739675 +0000 UTC m=+698.330321726" watchObservedRunningTime="2026-11-03 10:41:09.918190324 +0000 UTC m=+700.123772378"
Nov 03 10:41:10.085718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:10.085644    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:41:10.085718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:10.085694    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 10:41:10.200683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:10.200654    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:41:10.397794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:10.397696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:11.025769 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:11.025742    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 10:41:11.085706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:11.085672    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:41:11.085896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:11.085734    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 10:41:11.254821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:11.254780    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:12.013033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:12.013000    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:12.033704 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:12.033681    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 10:41:12.086122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:12.086090    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:41:12.086273 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:12.086150    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 10:41:12.159360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:12.159323    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:12.601604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:12.601537    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:12.865776 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:12.865677    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:13.051803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.051773    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:13.089836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.089809    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:41:13.090148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.090130    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:41:13.090811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.090793    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 10:41:13.229370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.229331    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:13.373851 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.373804    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:13.518690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.518646    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:13.662920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.662877    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:13.801891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:13.801809    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:14.003279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:14.003236    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:14.587715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:14.587680    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:15.361444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:15.361403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:16.199878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:16.199843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:16.881188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:16.881157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:17.169481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:17.169388    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:17.183889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:17.183862    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:17.590381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:17.590346    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:18.188256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:18.188223    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:41:18.373888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:18.373853    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:18.687390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:18.687354    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:41:19.079272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:19.079183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:19.763347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:19.763311    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:20.466092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:20.466044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:21.167894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:21.167841    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:21.604926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:21.604899    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:22.045341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:22.045308    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:22.413318 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:41:22.413237    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:41:22.752622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:22.752556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:23.450218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:23.450183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:23.769092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:23.769013    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:41:24.165996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:24.165905    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:24.850481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:24.850439    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:24.988595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:24.988487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:25.460018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.459987    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"]
Nov 03 10:41:25.464852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.464834    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.472626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.472563    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"]
Nov 03 10:41:25.472996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.472976    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"openshift-service-ca.crt\""
Nov 03 10:41:25.473111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.472984    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"premium-w1-010232-kserve-self-signed-certs\""
Nov 03 10:41:25.473111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.473082    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"kube-root-ca.crt\""
Nov 03 10:41:25.473225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.473178    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"default-dockercfg-pfvjf\""
Nov 03 10:41:25.481244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.481224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:25.578421 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.578388    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-tmp-dir\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.578606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.578440    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-model-cache\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.578606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.578536    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-dshm\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.578700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.578636    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brkbf\" (UniqueName: \"kubernetes.io/projected/5795f519-3787-4f61-b640-6b828a6a56b8-kube-api-access-brkbf\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.578700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.578667    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-home\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.578778 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.578749    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5795f519-3787-4f61-b640-6b828a6a56b8-tls-certs\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.627767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.627732    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:25.694714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.693485    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-tmp-dir\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.694714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.693559    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-model-cache\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.694714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.693630    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-dshm\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.694714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.693685    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brkbf\" (UniqueName: \"kubernetes.io/projected/5795f519-3787-4f61-b640-6b828a6a56b8-kube-api-access-brkbf\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.694714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.693725    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-home\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.694714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.693790    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5795f519-3787-4f61-b640-6b828a6a56b8-tls-certs\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.694714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.694433    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-tmp-dir\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.706592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.706545    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-home\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.706994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.706975    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-model-cache\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.709547 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.709523    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5795f519-3787-4f61-b640-6b828a6a56b8-tls-certs\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.716223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.716134    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brkbf\" (UniqueName: \"kubernetes.io/projected/5795f519-3787-4f61-b640-6b828a6a56b8-kube-api-access-brkbf\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.716358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.716236    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-dshm\") pod \"premium-w1-010232-kserve-5c46c98877-v5wz9\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") " pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:25.779677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:25.779646    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:26.059240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:26.059206    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"]
Nov 03 10:41:26.061152 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:41:26.061127    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5795f519_3787_4f61_b640_6b828a6a56b8.slice/crio-ed7cba79e1d697c8df64016c13cc8a282dba0f32b0c23a608e80cdfd2fe5d708 WatchSource:0}: Error finding container ed7cba79e1d697c8df64016c13cc8a282dba0f32b0c23a608e80cdfd2fe5d708: Status 404 returned error can't find the container with id ed7cba79e1d697c8df64016c13cc8a282dba0f32b0c23a608e80cdfd2fe5d708
Nov 03 10:41:26.241434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:26.241401    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" event={"ID":"5795f519-3787-4f61-b640-6b828a6a56b8","Type":"ContainerStarted","Data":"744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854"}
Nov 03 10:41:26.241645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:26.241450    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" event={"ID":"5795f519-3787-4f61-b640-6b828a6a56b8","Type":"ContainerStarted","Data":"ed7cba79e1d697c8df64016c13cc8a282dba0f32b0c23a608e80cdfd2fe5d708"}
Nov 03 10:41:26.252684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:26.252627    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:26.264067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:26.263978    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podStartSLOduration=1.263963385 podStartE2EDuration="1.263963385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:26.258247703 +0000 UTC m=+716.463829755" watchObservedRunningTime="2026-11-03 10:41:26.263963385 +0000 UTC m=+716.469545500"
Nov 03 10:41:26.780764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:26.780734    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:26.843877 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:26.843749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:27.444357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:27.444325    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:28.130006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:28.129968    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:28.771252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:28.771219    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:29.571539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:29.571504    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:30.157009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:30.156951    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:30.808122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:30.808054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:31.445344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.444851    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:31.501227 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.501198    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"]
Nov 03 10:41:31.517794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.517758    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"]
Nov 03 10:41:31.517960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.517899    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.522655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.522635    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"premium-w4-d5d3c3-kserve-self-signed-certs\""
Nov 03 10:41:31.590463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.590426    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:31.678011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.677975    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-model-cache\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.678011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.678022    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vccx5\" (UniqueName: \"kubernetes.io/projected/10f80020-b416-45e1-b616-ee44e0668e45-kube-api-access-vccx5\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.678301 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.678134    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-home\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.678301 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.678212    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80020-b416-45e1-b616-ee44e0668e45-tls-certs\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.678430 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.678399    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-tmp-dir\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.678497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.678450    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-dshm\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.779760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.779683    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-home\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.779760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.779735    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80020-b416-45e1-b616-ee44e0668e45-tls-certs\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.779957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.779876    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-tmp-dir\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.779957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.779908    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-dshm\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.780077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.779956    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-model-cache\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.780077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.779991    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vccx5\" (UniqueName: \"kubernetes.io/projected/10f80020-b416-45e1-b616-ee44e0668e45-kube-api-access-vccx5\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.780186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.780146    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-home\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.780307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.780289    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-tmp-dir\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.780423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.780393    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-model-cache\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.782294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.782271    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-dshm\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.782386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.782293    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80020-b416-45e1-b616-ee44e0668e45-tls-certs\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.787687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.787665    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vccx5\" (UniqueName: \"kubernetes.io/projected/10f80020-b416-45e1-b616-ee44e0668e45-kube-api-access-vccx5\") pod \"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") " pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:31.834112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:31.834090    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:32.110010 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:41:32.109963    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10f80020_b416_45e1_b616_ee44e0668e45.slice/crio-04a07cf3391143c5479b02ed269137df39bd6341701d1e5d25fc52a5e2522a25 WatchSource:0}: Error finding container 04a07cf3391143c5479b02ed269137df39bd6341701d1e5d25fc52a5e2522a25: Status 404 returned error can't find the container with id 04a07cf3391143c5479b02ed269137df39bd6341701d1e5d25fc52a5e2522a25
Nov 03 10:41:32.110738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:32.110705    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"]
Nov 03 10:41:32.144110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:32.143860    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:32.282010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:32.281973    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" event={"ID":"10f80020-b416-45e1-b616-ee44e0668e45","Type":"ContainerStarted","Data":"8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c"}
Nov 03 10:41:32.282166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:32.282022    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" event={"ID":"10f80020-b416-45e1-b616-ee44e0668e45","Type":"ContainerStarted","Data":"04a07cf3391143c5479b02ed269137df39bd6341701d1e5d25fc52a5e2522a25"}
Nov 03 10:41:32.299585 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:32.299537    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podStartSLOduration=1.299524299 podStartE2EDuration="1.299524299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:32.297681204 +0000 UTC m=+722.503263256" watchObservedRunningTime="2026-11-03 10:41:32.299524299 +0000 UTC m=+722.505106349"
Nov 03 10:41:32.761539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:32.761512    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:32.834879 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:32.834846    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:32.944932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:32.944811    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:33.647343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:33.647307    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:34.369719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:34.369672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:34.966735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:34.966703    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:35.463696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:35.463661    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:41:35.669781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:35.669751    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:35.780086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:35.780001    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:35.793639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:35.793600    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:36.330044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:36.330008    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:41:36.548599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:36.542849    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:37.169278 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:37.167733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:37.820594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:37.820025    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:38.467428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:38.467373    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:39.248881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:39.248814    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:39.963990 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:39.963944    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:40.640825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:40.640790    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:40.826123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:40.826092    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:41.213447 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:41.213410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:41.367178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:41.366027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:41.835234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:41.835205    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:41.847454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:41.847432    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:42.049907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:42.049872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:42.361702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:42.361672    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:41:42.753433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:42.753402    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:43.541824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:43.541471    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:44.323504 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:44.323469    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:45.269345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:45.269315    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:45.939474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:45.939284    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:46.249988 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.249897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:46.470237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.470197    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"]
Nov 03 10:41:46.473826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.473802    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.481361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.481337    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"kube-root-ca.crt\""
Nov 03 10:41:46.481841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.481823    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"default-dockercfg-6dbqs\""
Nov 03 10:41:46.482120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.482102    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"openshift-service-ca.crt\""
Nov 03 10:41:46.483985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.483962    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"premium-w2-e78c33-kserve-self-signed-certs\""
Nov 03 10:41:46.484495 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.484474    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"]
Nov 03 10:41:46.535753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.535668    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-dshm\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.535753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.535721    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-model-cache\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.535966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.535854    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhgx\" (UniqueName: \"kubernetes.io/projected/4d3c3349-0d6e-4277-820c-f008f68c716d-kube-api-access-hmhgx\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.535966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.535912    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-home\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.535966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.535951    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-tmp-dir\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.536073 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.535978    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3c3349-0d6e-4277-820c-f008f68c716d-tls-certs\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.571107 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.571079    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:46.636958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.636924    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-home\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.637102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.636987    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-tmp-dir\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.637102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.637023    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3c3349-0d6e-4277-820c-f008f68c716d-tls-certs\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.637215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.637110    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-dshm\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.637215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.637146    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-model-cache\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.637215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.637208    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhgx\" (UniqueName: \"kubernetes.io/projected/4d3c3349-0d6e-4277-820c-f008f68c716d-kube-api-access-hmhgx\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.637374 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.637335    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-home\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.637516 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.637482    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-model-cache\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.637648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.637494    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-tmp-dir\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.639417 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.639398    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-dshm\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.639599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.639564    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3c3349-0d6e-4277-820c-f008f68c716d-tls-certs\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.645134 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.645112    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhgx\" (UniqueName: \"kubernetes.io/projected/4d3c3349-0d6e-4277-820c-f008f68c716d-kube-api-access-hmhgx\") pod \"premium-w2-e78c33-kserve-57b77754df-xcpxq\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") " pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:46.715069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.714901    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:46.789915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:46.789844    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:47.075566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:47.075542    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"]
Nov 03 10:41:47.077506 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:41:47.077468    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d3c3349_0d6e_4277_820c_f008f68c716d.slice/crio-5e8eea1fc1f1fdb0207b073400f88d70b6e6046e8f5724f3b30249a128082e42 WatchSource:0}: Error finding container 5e8eea1fc1f1fdb0207b073400f88d70b6e6046e8f5724f3b30249a128082e42: Status 404 returned error can't find the container with id 5e8eea1fc1f1fdb0207b073400f88d70b6e6046e8f5724f3b30249a128082e42
Nov 03 10:41:47.385861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:47.385774    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" event={"ID":"4d3c3349-0d6e-4277-820c-f008f68c716d","Type":"ContainerStarted","Data":"26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42"}
Nov 03 10:41:47.385861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:47.385821    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" event={"ID":"4d3c3349-0d6e-4277-820c-f008f68c716d","Type":"ContainerStarted","Data":"5e8eea1fc1f1fdb0207b073400f88d70b6e6046e8f5724f3b30249a128082e42"}
Nov 03 10:41:47.404773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:47.404690    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podStartSLOduration=1.40467224 podStartE2EDuration="1.40467224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:47.400928662 +0000 UTC m=+737.606510713" watchObservedRunningTime="2026-11-03 10:41:47.40467224 +0000 UTC m=+737.610254293"
Nov 03 10:41:47.494363 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:47.494328    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:47.693763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:47.693734    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:41:47.789991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:47.789959    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:48.484465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:48.484424    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:49.621115 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:49.621077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:50.706604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:50.706556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:51.654195 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:51.654162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:52.428460 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:41:52.428422    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:41:52.494533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:52.494498    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:53.264055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:53.264001    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:53.941360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:53.941297    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:54.602285 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:54.602253    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:55.346180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:55.346145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:56.172235 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:56.172204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:56.790671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:56.790640    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:56.807695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:56.807671    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:56.923601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:56.923219    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:57.470164 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:57.470138    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:41:57.682906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:57.682874    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:58.356509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:58.356401    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:59.037434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:59.037386    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:41:59.755487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:41:59.755448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:00.533905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:00.532279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:01.054881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:01.054846    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:42:01.278278 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:01.278244    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:02.271066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:02.271030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:03.019428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:03.019398    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:03.680820 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:03.680732    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:04.324476 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:04.324443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:05.021361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.021325    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"]
Nov 03 10:42:05.023818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.023791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:05.026802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.026783    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.030246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.030219    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:42:05.031468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.031439    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:42:05.035267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.035244    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"]
Nov 03 10:42:05.054188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.054166    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.054316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.054210    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdws\" (UniqueName: \"kubernetes.io/projected/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-kube-api-access-lwdws\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.054394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.054370    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.155647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.155609    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.155835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.155673    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdws\" (UniqueName: \"kubernetes.io/projected/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-kube-api-access-lwdws\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.155835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.155739    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.156480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.156397    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.158237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.158208    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.163619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.163591    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdws\" (UniqueName: \"kubernetes.io/projected/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-kube-api-access-lwdws\") pod \"maas-api-revoke-sub-16b3eabe-zbznk\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.343464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.343381    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:05.619887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.619864    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"]
Nov 03 10:42:05.621459 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:42:05.621437    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe18c3b_7ae3_4b84_8647_a44c517fbccb.slice/crio-e687b4d0c02367e83dad7e3871a31b13e5380ab4f7ad59b071d2ae3b318fea0f WatchSource:0}: Error finding container e687b4d0c02367e83dad7e3871a31b13e5380ab4f7ad59b071d2ae3b318fea0f: Status 404 returned error can't find the container with id e687b4d0c02367e83dad7e3871a31b13e5380ab4f7ad59b071d2ae3b318fea0f
Nov 03 10:42:05.656231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:05.656201    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:06.371920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:06.371876    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:06.529878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:06.529850    5817 generic.go:356] "Generic (PLEG): container finished" podID="fbe18c3b-7ae3-4b84-8647-a44c517fbccb" containerID="d62ec2b2c39aad0c596d3cbe190be125ce86a87e6d1e60af7d54435ade03fd07" exitCode=0
Nov 03 10:42:06.530051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:06.529941    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk" event={"ID":"fbe18c3b-7ae3-4b84-8647-a44c517fbccb","Type":"ContainerDied","Data":"d62ec2b2c39aad0c596d3cbe190be125ce86a87e6d1e60af7d54435ade03fd07"}
Nov 03 10:42:06.530051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:06.529997    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk" event={"ID":"fbe18c3b-7ae3-4b84-8647-a44c517fbccb","Type":"ContainerStarted","Data":"e687b4d0c02367e83dad7e3871a31b13e5380ab4f7ad59b071d2ae3b318fea0f"}
Nov 03 10:42:07.053787 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.053751    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:07.651387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.651276    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:07.694265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.694243    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:07.785091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.785062    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-kube-api-access-lwdws\" (UniqueName: \"kubernetes.io/projected/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-kube-api-access-lwdws\") pod \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") "
Nov 03 10:42:07.785249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.785154    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-openshift-ca\") pod \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") "
Nov 03 10:42:07.785249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.785192    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-xks-ca\") pod \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\" (UID: \"fbe18c3b-7ae3-4b84-8647-a44c517fbccb\") "
Nov 03 10:42:07.785566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.785540    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-openshift-ca" pod "fbe18c3b-7ae3-4b84-8647-a44c517fbccb" (UID: "fbe18c3b-7ae3-4b84-8647-a44c517fbccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:42:07.787327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.787300    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-xks-ca" pod "fbe18c3b-7ae3-4b84-8647-a44c517fbccb" (UID: "fbe18c3b-7ae3-4b84-8647-a44c517fbccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:42:07.787428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.787406    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-kube-api-access-lwdws" pod "fbe18c3b-7ae3-4b84-8647-a44c517fbccb" (UID: "fbe18c3b-7ae3-4b84-8647-a44c517fbccb"). InnerVolumeSpecName "kube-api-access-lwdws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:42:07.886399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.886374    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:07.886399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.886400    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwdws\" (UniqueName: \"kubernetes.io/projected/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-kube-api-access-lwdws\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:07.886562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:07.886411    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe18c3b-7ae3-4b84-8647-a44c517fbccb-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:08.239530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:08.239491    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:08.545047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:08.544969    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"
Nov 03 10:42:08.545047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:08.544999    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk" event={"ID":"fbe18c3b-7ae3-4b84-8647-a44c517fbccb","Type":"ContainerDied","Data":"e687b4d0c02367e83dad7e3871a31b13e5380ab4f7ad59b071d2ae3b318fea0f"}
Nov 03 10:42:08.545235 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:08.545066    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e687b4d0c02367e83dad7e3871a31b13e5380ab4f7ad59b071d2ae3b318fea0f"
Nov 03 10:42:08.865304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:08.865208    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:09.755856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:09.755823    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:10.469415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:10.469376    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:11.160451 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:11.160418    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:11.875729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:11.875697    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:12.347144 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.347116    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"]
Nov 03 10:42:12.347592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.347557    5817 cpu_manager.go:414] "containerMap: removing container" podUID="fbe18c3b-7ae3-4b84-8647-a44c517fbccb" containerName="revoke-keys"
Nov 03 10:42:12.347592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.347586    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe18c3b-7ae3-4b84-8647-a44c517fbccb" containerName="revoke-keys"
Nov 03 10:42:12.347721 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.347711    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe18c3b-7ae3-4b84-8647-a44c517fbccb" containerName="revoke-keys"
Nov 03 10:42:12.351096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.351066    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.359815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.359774    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d5d3c3-openshift-default-dockercfg-wbpds\""
Nov 03 10:42:12.364163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.364045    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"]
Nov 03 10:42:12.444298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444266    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799609a-512e-4add-8320-979ebf65e473-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.444467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444340    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.444467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444414    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.444467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444461    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.444583 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444482    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.444583 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444551    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.444652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444597    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxgb\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-kube-api-access-5sxgb\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.444702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444678    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.444737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.444723    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799609a-512e-4add-8320-979ebf65e473-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.514836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.514750    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:12.546326 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546291    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxgb\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-kube-api-access-5sxgb\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546360    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546401    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799609a-512e-4add-8320-979ebf65e473-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546426    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799609a-512e-4add-8320-979ebf65e473-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546505    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546547    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546613    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546639    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546777    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546867    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.546893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.546888    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.547206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.547181    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.547302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.547246    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.547471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.547454    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799609a-512e-4add-8320-979ebf65e473-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.549058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.549035    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.549290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.549269    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799609a-512e-4add-8320-979ebf65e473-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.554179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.554156    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.554179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.554175    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxgb\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-kube-api-access-5sxgb\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.670069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.669990    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:12.957801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.957773    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"]
Nov 03 10:42:12.958429 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:42:12.958402    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2799609a_512e_4add_8320_979ebf65e473.slice/crio-da98ce08f3441385dcaef2f1d3cfec6cece8f00534c87cc395bc161a86aba043 WatchSource:0}: Error finding container da98ce08f3441385dcaef2f1d3cfec6cece8f00534c87cc395bc161a86aba043: Status 404 returned error can't find the container with id da98ce08f3441385dcaef2f1d3cfec6cece8f00534c87cc395bc161a86aba043
Nov 03 10:42:12.960686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.960656    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:42:12.960752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.960735    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:42:12.960789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:12.960764    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:42:13.158802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:13.158762    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:13.585552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:13.585517    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" event={"ID":"2799609a-512e-4add-8320-979ebf65e473","Type":"ContainerStarted","Data":"852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc"}
Nov 03 10:42:13.585729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:13.585605    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" event={"ID":"2799609a-512e-4add-8320-979ebf65e473","Type":"ContainerStarted","Data":"da98ce08f3441385dcaef2f1d3cfec6cece8f00534c87cc395bc161a86aba043"}
Nov 03 10:42:13.646866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:13.646814    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podStartSLOduration=1.646800765 podStartE2EDuration="1.646800765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:13.643439014 +0000 UTC m=+763.849021064" watchObservedRunningTime="2026-11-03 10:42:13.646800765 +0000 UTC m=+763.852382816"
Nov 03 10:42:13.671050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:13.671022    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:13.836360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:13.836266    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:14.450602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:14.450469    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:14.477335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:14.477306    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:42:14.672220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:14.672187    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 10:42:14.672432 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:14.672248    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 10:42:15.077097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:15.077064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:15.661602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:15.661543    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:15.671117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:15.671082    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 10:42:15.671293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:15.671155    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 10:42:16.261695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:16.261666    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:16.670594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:16.670490    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 10:42:16.670594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:16.670547    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 10:42:16.962384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:16.962303    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:17.562369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:17.561226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:17.670878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:17.670845    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 10:42:17.671280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:17.670899    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 10:42:17.953047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:17.953008    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"]
Nov 03 10:42:17.956983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:17.956960    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:17.961648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:17.961602    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:42:17.962934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:17.962900    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:42:17.964667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:17.964649    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"]
Nov 03 10:42:18.028982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.028948    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.029173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.029062    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.029173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.029106    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/5e74d07d-656f-420e-997e-2b28d0aaade9-kube-api-access-mkwc6\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.130006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.129976    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.130178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.130062    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.130178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.130097    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/5e74d07d-656f-420e-997e-2b28d0aaade9-kube-api-access-mkwc6\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.130826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.130748    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.132532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.132511    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.138305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.138279    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/5e74d07d-656f-420e-997e-2b28d0aaade9-kube-api-access-mkwc6\") pod \"maas-api-revoke-sub-52692ea1-qfpz8\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.259362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.259188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:18.271949 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.271924    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:18.576112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.576077    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"]
Nov 03 10:42:18.578348 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:42:18.578319    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e74d07d_656f_420e_997e_2b28d0aaade9.slice/crio-20754cf6c4dbfad3e8cbaae243bb9521989d8c25b804e51068d375f6bd6da1b8 WatchSource:0}: Error finding container 20754cf6c4dbfad3e8cbaae243bb9521989d8c25b804e51068d375f6bd6da1b8: Status 404 returned error can't find the container with id 20754cf6c4dbfad3e8cbaae243bb9521989d8c25b804e51068d375f6bd6da1b8
Nov 03 10:42:18.645787 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.645746    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8" event={"ID":"5e74d07d-656f-420e-997e-2b28d0aaade9","Type":"ContainerStarted","Data":"20754cf6c4dbfad3e8cbaae243bb9521989d8c25b804e51068d375f6bd6da1b8"}
Nov 03 10:42:18.671130 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.671105    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 10:42:18.671459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:18.671156    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 10:42:19.038480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:19.038445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:19.640057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:19.640020    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:19.659545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:19.659515    5817 generic.go:356] "Generic (PLEG): container finished" podID="5e74d07d-656f-420e-997e-2b28d0aaade9" containerID="618f736258153a0b23e52fca972c2a2fda4297574b157669fc78c9337667b5e9" exitCode=0
Nov 03 10:42:19.659773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:19.659693    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8" event={"ID":"5e74d07d-656f-420e-997e-2b28d0aaade9","Type":"ContainerDied","Data":"618f736258153a0b23e52fca972c2a2fda4297574b157669fc78c9337667b5e9"}
Nov 03 10:42:19.795644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:19.795540    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:19.796773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:19.796750    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:19.797810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:19.797788    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 10:42:20.337849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.337487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:20.816003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.815979    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:20.949106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.949071    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:20.969079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.969048    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-xks-ca\") pod \"5e74d07d-656f-420e-997e-2b28d0aaade9\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") "
Nov 03 10:42:20.969265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.969112    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-openshift-ca\") pod \"5e74d07d-656f-420e-997e-2b28d0aaade9\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") "
Nov 03 10:42:20.969265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.969147    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e74d07d-656f-420e-997e-2b28d0aaade9-kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/5e74d07d-656f-420e-997e-2b28d0aaade9-kube-api-access-mkwc6\") pod \"5e74d07d-656f-420e-997e-2b28d0aaade9\" (UID: \"5e74d07d-656f-420e-997e-2b28d0aaade9\") "
Nov 03 10:42:20.969506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.969485    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-openshift-ca" pod "5e74d07d-656f-420e-997e-2b28d0aaade9" (UID: "5e74d07d-656f-420e-997e-2b28d0aaade9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:42:20.969664 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.969648    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:20.971383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.971351    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e74d07d-656f-420e-997e-2b28d0aaade9-kube-api-access-mkwc6" pod "5e74d07d-656f-420e-997e-2b28d0aaade9" (UID: "5e74d07d-656f-420e-997e-2b28d0aaade9"). InnerVolumeSpecName "kube-api-access-mkwc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:42:20.971496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:20.971424    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-xks-ca" pod "5e74d07d-656f-420e-997e-2b28d0aaade9" (UID: "5e74d07d-656f-420e-997e-2b28d0aaade9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:42:21.070404 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:21.070374    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/5e74d07d-656f-420e-997e-2b28d0aaade9-kube-api-access-mkwc6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:21.070404 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:21.070398    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e74d07d-656f-420e-997e-2b28d0aaade9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:21.673143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:21.673116    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"
Nov 03 10:42:21.673322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:21.673154    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8" event={"ID":"5e74d07d-656f-420e-997e-2b28d0aaade9","Type":"ContainerDied","Data":"20754cf6c4dbfad3e8cbaae243bb9521989d8c25b804e51068d375f6bd6da1b8"}
Nov 03 10:42:21.673322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:21.673204    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20754cf6c4dbfad3e8cbaae243bb9521989d8c25b804e51068d375f6bd6da1b8"
Nov 03 10:42:21.698016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:21.697988    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:22.433755 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:42:22.433704    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:42:22.627377 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:22.627339    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:23.355237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:23.355206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:23.984873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:23.984834    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:24.568741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:24.568702    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:25.153869 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:25.153832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:25.850800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:25.850762    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:26.343332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.343296    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"]
Nov 03 10:42:26.343899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.343883    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5e74d07d-656f-420e-997e-2b28d0aaade9" containerName="revoke-keys"
Nov 03 10:42:26.343953 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.343899    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e74d07d-656f-420e-997e-2b28d0aaade9" containerName="revoke-keys"
Nov 03 10:42:26.344021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.344009    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e74d07d-656f-420e-997e-2b28d0aaade9" containerName="revoke-keys"
Nov 03 10:42:26.347601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.347566    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.352088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.352063    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:42:26.352259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.352163    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:42:26.354932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.354910    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"]
Nov 03 10:42:26.430841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.430801    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbl4q\" (UniqueName: \"kubernetes.io/projected/b0c88769-92c7-468f-965f-a790e4a17241-kube-api-access-fbl4q\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.431022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.430865    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.431022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.430960    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.532226 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.532194    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbl4q\" (UniqueName: \"kubernetes.io/projected/b0c88769-92c7-468f-965f-a790e4a17241-kube-api-access-fbl4q\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.532387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.532247    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.532387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.532288    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.533159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.533106    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.534833 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.534815    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.540123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.540098    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbl4q\" (UniqueName: \"kubernetes.io/projected/b0c88769-92c7-468f-965f-a790e4a17241-kube-api-access-fbl4q\") pod \"maas-api-revoke-sub-dbe97acb-sdzkl\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.556246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.556117    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:26.668592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.668499    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:26.945497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:26.945473    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"]
Nov 03 10:42:26.947326 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:42:26.947300    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c88769_92c7_468f_965f_a790e4a17241.slice/crio-cb8c52ceae778b319e97240752cc253fd48888f1af9be81c767b4acd7b873e32 WatchSource:0}: Error finding container cb8c52ceae778b319e97240752cc253fd48888f1af9be81c767b4acd7b873e32: Status 404 returned error can't find the container with id cb8c52ceae778b319e97240752cc253fd48888f1af9be81c767b4acd7b873e32
Nov 03 10:42:27.169535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:27.169499    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:27.715665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:27.715627    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0c88769-92c7-468f-965f-a790e4a17241" containerID="c04e832fe02ec68555715fc6407784e58c1de170968e12e86a6df05b5768e7cd" exitCode=0
Nov 03 10:42:27.716011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:27.715666    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl" event={"ID":"b0c88769-92c7-468f-965f-a790e4a17241","Type":"ContainerDied","Data":"c04e832fe02ec68555715fc6407784e58c1de170968e12e86a6df05b5768e7cd"}
Nov 03 10:42:27.716011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:27.715725    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl" event={"ID":"b0c88769-92c7-468f-965f-a790e4a17241","Type":"ContainerStarted","Data":"cb8c52ceae778b319e97240752cc253fd48888f1af9be81c767b4acd7b873e32"}
Nov 03 10:42:27.855722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:27.855691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:28.462440 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:28.462397    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:28.877098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:28.877074    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:28.960123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:28.960072    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c88769-92c7-468f-965f-a790e4a17241-kube-api-access-fbl4q\" (UniqueName: \"kubernetes.io/projected/b0c88769-92c7-468f-965f-a790e4a17241-kube-api-access-fbl4q\") pod \"b0c88769-92c7-468f-965f-a790e4a17241\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") "
Nov 03 10:42:28.960296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:28.960183    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-xks-ca\") pod \"b0c88769-92c7-468f-965f-a790e4a17241\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") "
Nov 03 10:42:28.960296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:28.960265    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-openshift-ca\") pod \"b0c88769-92c7-468f-965f-a790e4a17241\" (UID: \"b0c88769-92c7-468f-965f-a790e4a17241\") "
Nov 03 10:42:28.960852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:28.960804    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-openshift-ca" pod "b0c88769-92c7-468f-965f-a790e4a17241" (UID: "b0c88769-92c7-468f-965f-a790e4a17241"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:42:28.962935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:28.962849    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c88769-92c7-468f-965f-a790e4a17241-kube-api-access-fbl4q" pod "b0c88769-92c7-468f-965f-a790e4a17241" (UID: "b0c88769-92c7-468f-965f-a790e4a17241"). InnerVolumeSpecName "kube-api-access-fbl4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:42:28.963969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:28.963296    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-xks-ca" pod "b0c88769-92c7-468f-965f-a790e4a17241" (UID: "b0c88769-92c7-468f-965f-a790e4a17241"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:42:29.063281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.063174    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:29.063281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.063218    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c88769-92c7-468f-965f-a790e4a17241-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:29.063281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.063260    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbl4q\" (UniqueName: \"kubernetes.io/projected/b0c88769-92c7-468f-965f-a790e4a17241-kube-api-access-fbl4q\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:29.065635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.065606    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:29.165433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.165396    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:42:29.201456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.201417    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"]
Nov 03 10:42:29.202015 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.201995    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b0c88769-92c7-468f-965f-a790e4a17241" containerName="revoke-keys"
Nov 03 10:42:29.202125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.202019    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c88769-92c7-468f-965f-a790e4a17241" containerName="revoke-keys"
Nov 03 10:42:29.202201 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.202182    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c88769-92c7-468f-965f-a790e4a17241" containerName="revoke-keys"
Nov 03 10:42:29.205633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.205614    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.212597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.212537    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"]
Nov 03 10:42:29.368010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.367914    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.368010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.367958    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptmn\" (UniqueName: \"kubernetes.io/projected/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-kube-api-access-8ptmn\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.368010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.367993    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.431789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.431749    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"]
Nov 03 10:42:29.436378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.436354    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.441731 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.441511    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f78621-openshift-default-dockercfg-2x24f\""
Nov 03 10:42:29.446262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.446236    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"]
Nov 03 10:42:29.469937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.469903    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.470118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.469956    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptmn\" (UniqueName: \"kubernetes.io/projected/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-kube-api-access-8ptmn\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.470118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.469984    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.470703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.470682    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.473242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.473214    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.482101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.481816    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptmn\" (UniqueName: \"kubernetes.io/projected/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-kube-api-access-8ptmn\") pod \"maas-api-revoke-sub-11264373-gvmd2\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.528254 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.528220    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:29.570720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.570680    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.570929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.570804    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.570929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.570883    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6vp\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-kube-api-access-xv6vp\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.571035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.570926    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.571035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.570967    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.571035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.570995    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.571178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.571059    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.571178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.571129    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.571178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.571167    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672205    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672250    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672286    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672567 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672331    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672567 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672392    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6vp\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-kube-api-access-xv6vp\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672567 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672428    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672567 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672557    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672623    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672692    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672758    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.672810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672773    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.673022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.672935    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.673079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.673022    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.673331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.673304    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.674797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.674779    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.674905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.674890    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.680131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.680110    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.680224 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.680193    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6vp\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-kube-api-access-xv6vp\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.763800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.762998    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:29.766606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.765631    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"
Nov 03 10:42:29.766606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.765645    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl" event={"ID":"b0c88769-92c7-468f-965f-a790e4a17241","Type":"ContainerDied","Data":"cb8c52ceae778b319e97240752cc253fd48888f1af9be81c767b4acd7b873e32"}
Nov 03 10:42:29.766606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.765711    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb8c52ceae778b319e97240752cc253fd48888f1af9be81c767b4acd7b873e32"
Nov 03 10:42:29.806799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.806540    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"]
Nov 03 10:42:29.809073 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:42:29.809045    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f658ba_4bbb_4ffd_9e31_e614e74f8e38.slice/crio-ec8da87e436988eae2e6dbef72c1c80eb3cf15096c3b26b8d3bcbaf983978dea WatchSource:0}: Error finding container ec8da87e436988eae2e6dbef72c1c80eb3cf15096c3b26b8d3bcbaf983978dea: Status 404 returned error can't find the container with id ec8da87e436988eae2e6dbef72c1c80eb3cf15096c3b26b8d3bcbaf983978dea
Nov 03 10:42:29.865219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:29.864991    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:30.031360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.031329    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:30.057909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.056468    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"]
Nov 03 10:42:30.057988 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:42:30.057906    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a65377_d9b1_4c74_9aa1_ec3c80aea824.slice/crio-22d1d1164433c93c244f7fb4e5feca8b6544f747e71a2716bcf635dbdd116b24 WatchSource:0}: Error finding container 22d1d1164433c93c244f7fb4e5feca8b6544f747e71a2716bcf635dbdd116b24: Status 404 returned error can't find the container with id 22d1d1164433c93c244f7fb4e5feca8b6544f747e71a2716bcf635dbdd116b24
Nov 03 10:42:30.061561 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.061528    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:42:30.061712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.061620    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:42:30.061712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.061651    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:42:30.569420 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.569323    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:30.774293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.774259    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" event={"ID":"b8a65377-d9b1-4c74-9aa1-ec3c80aea824","Type":"ContainerStarted","Data":"70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116"}
Nov 03 10:42:30.774468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.774357    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" event={"ID":"b8a65377-d9b1-4c74-9aa1-ec3c80aea824","Type":"ContainerStarted","Data":"22d1d1164433c93c244f7fb4e5feca8b6544f747e71a2716bcf635dbdd116b24"}
Nov 03 10:42:30.776124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.776089    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2" event={"ID":"13f658ba-4bbb-4ffd-9e31-e614e74f8e38","Type":"ContainerStarted","Data":"b70d6b7c46005325c43962c7618dde5abbd2f297cd36ebab1b2ff390d91fe2a7"}
Nov 03 10:42:30.776243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.776147    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2" event={"ID":"13f658ba-4bbb-4ffd-9e31-e614e74f8e38","Type":"ContainerStarted","Data":"ec8da87e436988eae2e6dbef72c1c80eb3cf15096c3b26b8d3bcbaf983978dea"}
Nov 03 10:42:30.803595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.796332    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podStartSLOduration=1.796312821 podStartE2EDuration="1.796312821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:30.793973696 +0000 UTC m=+780.999555749" watchObservedRunningTime="2026-11-03 10:42:30.796312821 +0000 UTC m=+781.001894874"
Nov 03 10:42:30.819798 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:30.819688    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2" podStartSLOduration=1.819674982 podStartE2EDuration="1.819674982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:30.81892975 +0000 UTC m=+781.024511802" watchObservedRunningTime="2026-11-03 10:42:30.819674982 +0000 UTC m=+781.025257038"
Nov 03 10:42:31.171505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:31.171389    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:31.764337 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:31.764310    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:31.766123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:31.766099    5817 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 10:42:31.766247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:31.766141    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 10:42:31.785294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:31.785270    5817 generic.go:356] "Generic (PLEG): container finished" podID="13f658ba-4bbb-4ffd-9e31-e614e74f8e38" containerID="b70d6b7c46005325c43962c7618dde5abbd2f297cd36ebab1b2ff390d91fe2a7" exitCode=0
Nov 03 10:42:31.785439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:31.785356    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2" event={"ID":"13f658ba-4bbb-4ffd-9e31-e614e74f8e38","Type":"ContainerDied","Data":"b70d6b7c46005325c43962c7618dde5abbd2f297cd36ebab1b2ff390d91fe2a7"}
Nov 03 10:42:31.857015 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:31.856980    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:32.445620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:32.445505    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:32.764210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:32.764123    5817 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 10:42:32.764210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:32.764180    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 10:42:33.022971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.022896    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:33.054032 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.053944    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:33.133147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.133113    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-kube-api-access-8ptmn\" (UniqueName: \"kubernetes.io/projected/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-kube-api-access-8ptmn\") pod \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") "
Nov 03 10:42:33.133361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.133199    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-xks-ca\") pod \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") "
Nov 03 10:42:33.133754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.133598    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-openshift-ca\") pod \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\" (UID: \"13f658ba-4bbb-4ffd-9e31-e614e74f8e38\") "
Nov 03 10:42:33.133995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.133968    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-openshift-ca" pod "13f658ba-4bbb-4ffd-9e31-e614e74f8e38" (UID: "13f658ba-4bbb-4ffd-9e31-e614e74f8e38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:42:33.134110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.134088    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:33.136006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.135988    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-xks-ca" pod "13f658ba-4bbb-4ffd-9e31-e614e74f8e38" (UID: "13f658ba-4bbb-4ffd-9e31-e614e74f8e38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:42:33.136168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.136137    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-kube-api-access-8ptmn" pod "13f658ba-4bbb-4ffd-9e31-e614e74f8e38" (UID: "13f658ba-4bbb-4ffd-9e31-e614e74f8e38"). InnerVolumeSpecName "kube-api-access-8ptmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:42:33.235547 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.235516    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:33.235547 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.235542    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ptmn\" (UniqueName: \"kubernetes.io/projected/13f658ba-4bbb-4ffd-9e31-e614e74f8e38-kube-api-access-8ptmn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:33.648558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.648523    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:33.764446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.764415    5817 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 10:42:33.764684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.764471    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 10:42:33.837156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.837124    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"
Nov 03 10:42:33.837329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.837148    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2" event={"ID":"13f658ba-4bbb-4ffd-9e31-e614e74f8e38","Type":"ContainerDied","Data":"ec8da87e436988eae2e6dbef72c1c80eb3cf15096c3b26b8d3bcbaf983978dea"}
Nov 03 10:42:33.837329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:33.837236    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec8da87e436988eae2e6dbef72c1c80eb3cf15096c3b26b8d3bcbaf983978dea"
Nov 03 10:42:34.262805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:34.262773    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:34.764557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:34.764521    5817 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 10:42:34.764972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:34.764600    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 10:42:34.853080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:34.853045    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:35.653049 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:35.653009    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:35.763830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:35.763794    5817 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 10:42:35.764012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:35.763841    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 10:42:36.236797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:36.236763    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:36.462430 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:36.462393    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:37.153796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:37.153685    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:37.455935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:37.455908    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:37.456457 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:37.456310    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:37.458181 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:37.458158    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 10:42:37.908297 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:37.908182    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:38.647162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:38.647128    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:39.348735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.348700    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:39.764159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.764127    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"]
Nov 03 10:42:39.764620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.764607    5817 cpu_manager.go:414] "containerMap: removing container" podUID="13f658ba-4bbb-4ffd-9e31-e614e74f8e38" containerName="revoke-keys"
Nov 03 10:42:39.764680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.764621    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f658ba-4bbb-4ffd-9e31-e614e74f8e38" containerName="revoke-keys"
Nov 03 10:42:39.764727 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.764720    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f658ba-4bbb-4ffd-9e31-e614e74f8e38" containerName="revoke-keys"
Nov 03 10:42:39.768163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.768148    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.773046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.773019    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:42:39.774396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.774375    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:42:39.777862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.777840    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"]
Nov 03 10:42:39.808265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.808231    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bplzr\" (UniqueName: \"kubernetes.io/projected/06e877ec-de2f-4359-ac50-925a16eabffe-kube-api-access-bplzr\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.808440 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.808317    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.808440 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.808414    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.909846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.909809    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bplzr\" (UniqueName: \"kubernetes.io/projected/06e877ec-de2f-4359-ac50-925a16eabffe-kube-api-access-bplzr\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.910005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.909867    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.910005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.909993    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.910821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.910791    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.912306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.912286    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:39.917541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:39.917519    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bplzr\" (UniqueName: \"kubernetes.io/projected/06e877ec-de2f-4359-ac50-925a16eabffe-kube-api-access-bplzr\") pod \"maas-api-revoke-sub-e520a03a-bgbx5\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:40.065970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:40.065867    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:40.083542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:40.083514    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:40.365030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:40.365003    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"]
Nov 03 10:42:40.366030 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:42:40.365993    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e877ec_de2f_4359_ac50_925a16eabffe.slice/crio-b9b87bb5e32b435fd5f6e2779d2dcfd9f9ba2f59672eb0c1840911e62602bb47 WatchSource:0}: Error finding container b9b87bb5e32b435fd5f6e2779d2dcfd9f9ba2f59672eb0c1840911e62602bb47: Status 404 returned error can't find the container with id b9b87bb5e32b435fd5f6e2779d2dcfd9f9ba2f59672eb0c1840911e62602bb47
Nov 03 10:42:40.887354 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:40.887315    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5" event={"ID":"06e877ec-de2f-4359-ac50-925a16eabffe","Type":"ContainerStarted","Data":"45621c29dc8ff9e4b3e0772046739235970cbc117e4ecf42ad2fa1fa2072d443"}
Nov 03 10:42:40.887884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:40.887369    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5" event={"ID":"06e877ec-de2f-4359-ac50-925a16eabffe","Type":"ContainerStarted","Data":"b9b87bb5e32b435fd5f6e2779d2dcfd9f9ba2f59672eb0c1840911e62602bb47"}
Nov 03 10:42:40.910249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:40.910187    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5" podStartSLOduration=1.910164714 podStartE2EDuration="1.910164714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:40.907934751 +0000 UTC m=+791.113516810" watchObservedRunningTime="2026-11-03 10:42:40.910164714 +0000 UTC m=+791.115746765"
Nov 03 10:42:40.914294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:40.914249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:41.258435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:41.258407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:41.563773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:41.563679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:41.942851 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:41.942813    5817 generic.go:356] "Generic (PLEG): container finished" podID="06e877ec-de2f-4359-ac50-925a16eabffe" containerID="45621c29dc8ff9e4b3e0772046739235970cbc117e4ecf42ad2fa1fa2072d443" exitCode=0
Nov 03 10:42:41.943733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:41.943700    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5" event={"ID":"06e877ec-de2f-4359-ac50-925a16eabffe","Type":"ContainerDied","Data":"45621c29dc8ff9e4b3e0772046739235970cbc117e4ecf42ad2fa1fa2072d443"}
Nov 03 10:42:42.348654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:42.342836    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:42.955467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:42.955194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:43.115992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.115962    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:43.156364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.156335    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-xks-ca\") pod \"06e877ec-de2f-4359-ac50-925a16eabffe\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") "
Nov 03 10:42:43.156523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.156494    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-openshift-ca\") pod \"06e877ec-de2f-4359-ac50-925a16eabffe\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") "
Nov 03 10:42:43.156589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.156525    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e877ec-de2f-4359-ac50-925a16eabffe-kube-api-access-bplzr\" (UniqueName: \"kubernetes.io/projected/06e877ec-de2f-4359-ac50-925a16eabffe-kube-api-access-bplzr\") pod \"06e877ec-de2f-4359-ac50-925a16eabffe\" (UID: \"06e877ec-de2f-4359-ac50-925a16eabffe\") "
Nov 03 10:42:43.156863 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.156833    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-openshift-ca" pod "06e877ec-de2f-4359-ac50-925a16eabffe" (UID: "06e877ec-de2f-4359-ac50-925a16eabffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:42:43.159117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.158953    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-xks-ca" pod "06e877ec-de2f-4359-ac50-925a16eabffe" (UID: "06e877ec-de2f-4359-ac50-925a16eabffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:42:43.159527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.159234    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e877ec-de2f-4359-ac50-925a16eabffe-kube-api-access-bplzr" pod "06e877ec-de2f-4359-ac50-925a16eabffe" (UID: "06e877ec-de2f-4359-ac50-925a16eabffe"). InnerVolumeSpecName "kube-api-access-bplzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:42:43.229847 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.229756    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:43.257610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.257560    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:43.257610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.257605    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bplzr\" (UniqueName: \"kubernetes.io/projected/06e877ec-de2f-4359-ac50-925a16eabffe-kube-api-access-bplzr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:43.257804 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.257624    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e877ec-de2f-4359-ac50-925a16eabffe-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:43.382815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.374313    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:43.473603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.472246    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"]
Nov 03 10:42:43.473603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.472915    5817 cpu_manager.go:414] "containerMap: removing container" podUID="06e877ec-de2f-4359-ac50-925a16eabffe" containerName="revoke-keys"
Nov 03 10:42:43.473603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.472931    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e877ec-de2f-4359-ac50-925a16eabffe" containerName="revoke-keys"
Nov 03 10:42:43.473603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.473058    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e877ec-de2f-4359-ac50-925a16eabffe" containerName="revoke-keys"
Nov 03 10:42:43.477371 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.476750    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.507736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.507700    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"]
Nov 03 10:42:43.561072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.561001    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jf5s\" (UniqueName: \"kubernetes.io/projected/f27b86fd-0819-4084-bfa1-f99dbd2d5884-kube-api-access-2jf5s\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.561219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.561141    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.561284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.561264    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.662801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.662769    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jf5s\" (UniqueName: \"kubernetes.io/projected/f27b86fd-0819-4084-bfa1-f99dbd2d5884-kube-api-access-2jf5s\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.662987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.662819    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.662987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.662917    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.663469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.663450    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.665438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.665405    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.671450 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.671420    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jf5s\" (UniqueName: \"kubernetes.io/projected/f27b86fd-0819-4084-bfa1-f99dbd2d5884-kube-api-access-2jf5s\") pod \"maas-api-revoke-sub-bc69b3fa-xxvnd\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.763716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.763637    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:43.820488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.820457    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:43.968372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.968290    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:43.982469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.982442    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"
Nov 03 10:42:43.984268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.984239    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5" event={"ID":"06e877ec-de2f-4359-ac50-925a16eabffe","Type":"ContainerDied","Data":"b9b87bb5e32b435fd5f6e2779d2dcfd9f9ba2f59672eb0c1840911e62602bb47"}
Nov 03 10:42:43.984447 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:43.984433    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b87bb5e32b435fd5f6e2779d2dcfd9f9ba2f59672eb0c1840911e62602bb47"
Nov 03 10:42:44.130653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:44.128609    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"]
Nov 03 10:42:44.661219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:44.661125    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:45.004597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:45.004042    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd" event={"ID":"f27b86fd-0819-4084-bfa1-f99dbd2d5884","Type":"ContainerStarted","Data":"03910994422c85882a9bc0f43922bbe70926a3d1c6bded0bf50c03d48d67869a"}
Nov 03 10:42:45.004597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:45.004100    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd" event={"ID":"f27b86fd-0819-4084-bfa1-f99dbd2d5884","Type":"ContainerStarted","Data":"393c1ef50d7a98b5bdd94e067589668b42a700946d98a5afeb3e3559fa1a2821"}
Nov 03 10:42:45.825614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:45.824895    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:46.019599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:46.018990    5817 generic.go:356] "Generic (PLEG): container finished" podID="f27b86fd-0819-4084-bfa1-f99dbd2d5884" containerID="03910994422c85882a9bc0f43922bbe70926a3d1c6bded0bf50c03d48d67869a" exitCode=0
Nov 03 10:42:46.019599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:46.019441    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd" event={"ID":"f27b86fd-0819-4084-bfa1-f99dbd2d5884","Type":"ContainerDied","Data":"03910994422c85882a9bc0f43922bbe70926a3d1c6bded0bf50c03d48d67869a"}
Nov 03 10:42:46.559772 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:46.559743    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:46.856749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:46.856657    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:47.179149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.179117    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:47.213124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.213093    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27b86fd-0819-4084-bfa1-f99dbd2d5884-kube-api-access-2jf5s\" (UniqueName: \"kubernetes.io/projected/f27b86fd-0819-4084-bfa1-f99dbd2d5884-kube-api-access-2jf5s\") pod \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") "
Nov 03 10:42:47.213297 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.213174    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-openshift-ca\") pod \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") "
Nov 03 10:42:47.213398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.213382    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-xks-ca\") pod \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\" (UID: \"f27b86fd-0819-4084-bfa1-f99dbd2d5884\") "
Nov 03 10:42:47.213600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.213541    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-openshift-ca" pod "f27b86fd-0819-4084-bfa1-f99dbd2d5884" (UID: "f27b86fd-0819-4084-bfa1-f99dbd2d5884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:42:47.213882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.213819    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:47.215654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.215549    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27b86fd-0819-4084-bfa1-f99dbd2d5884-kube-api-access-2jf5s" pod "f27b86fd-0819-4084-bfa1-f99dbd2d5884" (UID: "f27b86fd-0819-4084-bfa1-f99dbd2d5884"). InnerVolumeSpecName "kube-api-access-2jf5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:42:47.215920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.215900    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-xks-ca" pod "f27b86fd-0819-4084-bfa1-f99dbd2d5884" (UID: "f27b86fd-0819-4084-bfa1-f99dbd2d5884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:42:47.314938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.314902    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27b86fd-0819-4084-bfa1-f99dbd2d5884-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:47.314938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.314936    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jf5s\" (UniqueName: \"kubernetes.io/projected/f27b86fd-0819-4084-bfa1-f99dbd2d5884-kube-api-access-2jf5s\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:47.536448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:47.536404    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:48.034631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:48.034599    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd" event={"ID":"f27b86fd-0819-4084-bfa1-f99dbd2d5884","Type":"ContainerDied","Data":"393c1ef50d7a98b5bdd94e067589668b42a700946d98a5afeb3e3559fa1a2821"}
Nov 03 10:42:48.034810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:48.034639    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"
Nov 03 10:42:48.034810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:48.034647    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="393c1ef50d7a98b5bdd94e067589668b42a700946d98a5afeb3e3559fa1a2821"
Nov 03 10:42:48.151145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:48.151112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:48.820256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:48.820223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:49.590682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:49.590642    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:49.866556 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:49.866380    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:42:50.065111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.065073    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"]
Nov 03 10:42:50.066261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.066234    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f27b86fd-0819-4084-bfa1-f99dbd2d5884" containerName="revoke-keys"
Nov 03 10:42:50.066261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.066260    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27b86fd-0819-4084-bfa1-f99dbd2d5884" containerName="revoke-keys"
Nov 03 10:42:50.066518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.066503    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27b86fd-0819-4084-bfa1-f99dbd2d5884" containerName="revoke-keys"
Nov 03 10:42:50.071486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.071449    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.074892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.074871    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:42:50.075041 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.075019    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:42:50.075410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.075386    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"]
Nov 03 10:42:50.146598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.146495    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.146598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.146592    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xrb\" (UniqueName: \"kubernetes.io/projected/6fd5b757-fe1a-4755-8adf-30e8d630da0e-kube-api-access-d2xrb\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.146810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.146694    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.248307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.248259    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.248496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.248380    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xrb\" (UniqueName: \"kubernetes.io/projected/6fd5b757-fe1a-4755-8adf-30e8d630da0e-kube-api-access-d2xrb\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.248496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.248411    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.249045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.249020    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.251211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.251187    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.256458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.256433    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xrb\" (UniqueName: \"kubernetes.io/projected/6fd5b757-fe1a-4755-8adf-30e8d630da0e-kube-api-access-d2xrb\") pod \"maas-api-revoke-sub-7e5bda6a-587qp\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.353118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.353081    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:50.385825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.385798    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:50.663615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:50.663548    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"]
Nov 03 10:42:50.665868 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:42:50.665837    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd5b757_fe1a_4755_8adf_30e8d630da0e.slice/crio-2b2c043d277b9a9ebc11b6740ff71e7641e158cfe20dbe16081dd295d1afb3c8 WatchSource:0}: Error finding container 2b2c043d277b9a9ebc11b6740ff71e7641e158cfe20dbe16081dd295d1afb3c8: Status 404 returned error can't find the container with id 2b2c043d277b9a9ebc11b6740ff71e7641e158cfe20dbe16081dd295d1afb3c8
Nov 03 10:42:51.057549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:51.057508    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp" event={"ID":"6fd5b757-fe1a-4755-8adf-30e8d630da0e","Type":"ContainerStarted","Data":"2b2c043d277b9a9ebc11b6740ff71e7641e158cfe20dbe16081dd295d1afb3c8"}
Nov 03 10:42:51.234323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:51.233908    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:52.041651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:52.038214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:52.079646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:52.079592    5817 generic.go:356] "Generic (PLEG): container finished" podID="6fd5b757-fe1a-4755-8adf-30e8d630da0e" containerID="cfbe75e9f39ee53c30f252a9d4638e013c63692e8f60a8ec432aa8529011c43b" exitCode=0
Nov 03 10:42:52.080151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:52.079756    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp" event={"ID":"6fd5b757-fe1a-4755-8adf-30e8d630da0e","Type":"ContainerDied","Data":"cfbe75e9f39ee53c30f252a9d4638e013c63692e8f60a8ec432aa8529011c43b"}
Nov 03 10:42:52.473594 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:42:52.473400    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:42:52.921543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:52.915402    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:53.248597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.247829    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-c7n4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:42:53.248597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.247882    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" podUID="edf3ef4f-1840-4e68-9049-ec8a195d5e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:42:53.504103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.504041    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:53.669567 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.669540    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd5b757-fe1a-4755-8adf-30e8d630da0e-kube-api-access-d2xrb\" (UniqueName: \"kubernetes.io/projected/6fd5b757-fe1a-4755-8adf-30e8d630da0e-kube-api-access-d2xrb\") pod \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") "
Nov 03 10:42:53.669765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.669627    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-xks-ca\") pod \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") "
Nov 03 10:42:53.669765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.669734    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-openshift-ca\") pod \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\" (UID: \"6fd5b757-fe1a-4755-8adf-30e8d630da0e\") "
Nov 03 10:42:53.670147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.670112    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-openshift-ca" pod "6fd5b757-fe1a-4755-8adf-30e8d630da0e" (UID: "6fd5b757-fe1a-4755-8adf-30e8d630da0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:42:53.671966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.671940    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-xks-ca" pod "6fd5b757-fe1a-4755-8adf-30e8d630da0e" (UID: "6fd5b757-fe1a-4755-8adf-30e8d630da0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:42:53.672124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.672091    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd5b757-fe1a-4755-8adf-30e8d630da0e-kube-api-access-d2xrb" pod "6fd5b757-fe1a-4755-8adf-30e8d630da0e" (UID: "6fd5b757-fe1a-4755-8adf-30e8d630da0e"). InnerVolumeSpecName "kube-api-access-d2xrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:42:53.711274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.711232    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:53.771356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.771261    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2xrb\" (UniqueName: \"kubernetes.io/projected/6fd5b757-fe1a-4755-8adf-30e8d630da0e-kube-api-access-d2xrb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:53.771356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.771296    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:53.771356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:53.771312    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd5b757-fe1a-4755-8adf-30e8d630da0e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:54.115360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:54.115289    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"
Nov 03 10:42:54.115525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:54.115278    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp" event={"ID":"6fd5b757-fe1a-4755-8adf-30e8d630da0e","Type":"ContainerDied","Data":"2b2c043d277b9a9ebc11b6740ff71e7641e158cfe20dbe16081dd295d1afb3c8"}
Nov 03 10:42:54.115525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:54.115411    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b2c043d277b9a9ebc11b6740ff71e7641e158cfe20dbe16081dd295d1afb3c8"
Nov 03 10:42:54.350988 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:54.350841    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:55.144861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:55.144810    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:55.853205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:55.853167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:56.445472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:56.445429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:57.362611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:57.358617    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:58.184321 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:58.182753    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:59.064595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:59.064195    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:42:59.839308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:42:59.839273    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:00.545644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.545605    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"]
Nov 03 10:43:00.547236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.547212    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6fd5b757-fe1a-4755-8adf-30e8d630da0e" containerName="revoke-keys"
Nov 03 10:43:00.547422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.547407    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd5b757-fe1a-4755-8adf-30e8d630da0e" containerName="revoke-keys"
Nov 03 10:43:00.547773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.547759    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd5b757-fe1a-4755-8adf-30e8d630da0e" containerName="revoke-keys"
Nov 03 10:43:00.552419 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.552397    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.567482 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.567452    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"]
Nov 03 10:43:00.597645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.597605    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 10:43:00.598006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.597988    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-77sqt\""
Nov 03 10:43:00.600317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.600281    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 10:43:00.600932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.600913    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 10:43:00.601346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.601328    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 10:43:00.602225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.602209    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-49a4ab2b36b3\""
Nov 03 10:43:00.705779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.705738    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdmd\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-kube-api-access-7hdmd\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.705936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.705845    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-routing-overlay-volume\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.706093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.706070    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-plugins-config-volume\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.706152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.706137    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-provider-credentials\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.772190 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.772116    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:00.806993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.806912    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-plugins-config-volume\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.806993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.806980    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-provider-credentials\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.807211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.807028    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdmd\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-kube-api-access-7hdmd\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.807211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.807068    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-routing-overlay-volume\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.807805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.807777    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-routing-overlay-volume\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.807931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.807857    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-plugins-config-volume\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.809668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.809635    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-provider-credentials\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.814552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.814531    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdmd\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-kube-api-access-7hdmd\") pod \"payload-processing-external-model-8bb6cf694-ttfc8\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") " pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:00.947131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:00.947101    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:01.230762 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:01.230729    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"]
Nov 03 10:43:01.232187 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:43:01.232163    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eef0755_9967_45d0_af8e_04bb96178702.slice/crio-4840daef7fe8d90a1b45de3c7e54b0d31331b4728e0efac2a0df3253ed9f5726 WatchSource:0}: Error finding container 4840daef7fe8d90a1b45de3c7e54b0d31331b4728e0efac2a0df3253ed9f5726: Status 404 returned error can't find the container with id 4840daef7fe8d90a1b45de3c7e54b0d31331b4728e0efac2a0df3253ed9f5726
Nov 03 10:43:01.369863 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:01.369835    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:01.965773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:01.965733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:02.143518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:02.143485    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:43:02.222519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:02.222441    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8" event={"ID":"7eef0755-9967-45d0-af8e-04bb96178702","Type":"ContainerStarted","Data":"cc1b755701c703ec2c2542e51a13838ae716239773829f98fdf9972f02a3ebff"}
Nov 03 10:43:02.222519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:02.222492    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8" event={"ID":"7eef0755-9967-45d0-af8e-04bb96178702","Type":"ContainerStarted","Data":"4840daef7fe8d90a1b45de3c7e54b0d31331b4728e0efac2a0df3253ed9f5726"}
Nov 03 10:43:02.222704 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:02.222593    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:02.238372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:02.238323    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8" podStartSLOduration=2.238310523 podStartE2EDuration="2.238310523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:02.237174697 +0000 UTC m=+812.442756748" watchObservedRunningTime="2026-11-03 10:43:02.238310523 +0000 UTC m=+812.443892574"
Nov 03 10:43:02.598177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:02.598075    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:03.340983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:03.340949    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:03.936399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:03.936083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:04.264634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:04.264542    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:04.749508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:04.749473    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:05.345292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:05.345259    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:05.963888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:05.963527    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:06.767947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:06.767904    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:07.532955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:07.532361    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:08.365427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:08.365392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:09.182442 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.182397    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:09.484977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.484880    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"]
Nov 03 10:43:09.497240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.495824    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.500453 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.500427    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"]
Nov 03 10:43:09.524820 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.524780    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.525127 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.525105    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.525367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.525349    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.525623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.525605    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.525853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.525834    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.526078 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.526061    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.526312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.526293    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.526832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.526792    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.527018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.526999    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmtf\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-kube-api-access-hsmtf\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628435    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628482    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmtf\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-kube-api-access-hsmtf\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628555    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628621    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628669    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628727    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628777    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628834    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.629362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.628860    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.631139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.630467    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.631139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.630485    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.631139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.630744    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.631139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.631052    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.632410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.632388    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.633314 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.633289    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.634285 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.634243    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.639477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.639434    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmtf\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-kube-api-access-hsmtf\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.640231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.640208    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-4s98l\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:09.849097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:09.849025    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:10.173996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:10.173868    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:10.231592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:10.230177    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"]
Nov 03 10:43:10.263437 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:43:10.263394    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb23b8b1_e31c_4066_8e09_57f80e621e6d.slice/crio-0f994c978848b821f1703de199dfe40114749863a0a3cf787dbe08c2c02d8566 WatchSource:0}: Error finding container 0f994c978848b821f1703de199dfe40114749863a0a3cf787dbe08c2c02d8566: Status 404 returned error can't find the container with id 0f994c978848b821f1703de199dfe40114749863a0a3cf787dbe08c2c02d8566
Nov 03 10:43:10.267037 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:10.267000    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:10.267159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:10.267082    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:10.267159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:10.267124    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:10.329992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:10.329957    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" event={"ID":"fb23b8b1-e31c-4066-8e09-57f80e621e6d","Type":"ContainerStarted","Data":"0f994c978848b821f1703de199dfe40114749863a0a3cf787dbe08c2c02d8566"}
Nov 03 10:43:11.005491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:11.005455    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:11.471875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:11.471838    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" event={"ID":"fb23b8b1-e31c-4066-8e09-57f80e621e6d","Type":"ContainerStarted","Data":"5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74"}
Nov 03 10:43:11.553784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:11.547373    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podStartSLOduration=2.547355105 podStartE2EDuration="2.547355105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:11.502525576 +0000 UTC m=+821.708107650" watchObservedRunningTime="2026-11-03 10:43:11.547355105 +0000 UTC m=+821.752937153"
Nov 03 10:43:11.853655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:11.852709    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:11.853655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:11.852867    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 10:43:11.853655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:11.852930    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 10:43:12.022157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.022122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:12.462138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.462094    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"]
Nov 03 10:43:12.466894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.466866    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.480067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.478620    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"]
Nov 03 10:43:12.626247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626212    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.626386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626273    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.626442 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626410    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.626557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626543    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.626625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626613    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.626661 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626634    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t657n\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-kube-api-access-t657n\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.626707 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626665    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80241a2b-be47-424f-a5e3-4c96ef8e3802-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.626797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626783    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.626912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.626900    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728161    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728263    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728319    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728362    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728447    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728495    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728512    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t657n\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-kube-api-access-t657n\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728534    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80241a2b-be47-424f-a5e3-4c96ef8e3802-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728553    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.728913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.728891    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.730398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.729270    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.730398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.729923    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80241a2b-be47-424f-a5e3-4c96ef8e3802-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.730398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.729935    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.730398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.730259    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.731768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.731725    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.733065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.733041    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.738004 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.737965    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t657n\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-kube-api-access-t657n\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.738350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.738330    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.742272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.742046    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:12.817881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.817838    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:12.850613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.850472    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 10:43:12.850613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:12.850532    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 10:43:13.126141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.125882    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"]
Nov 03 10:43:13.126908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.126875    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:13.127025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.126935    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:13.127025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.126975    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:13.242475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.241940    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:13.441758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.441615    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:13.525696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.525660    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" event={"ID":"80241a2b-be47-424f-a5e3-4c96ef8e3802","Type":"ContainerStarted","Data":"22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d"}
Nov 03 10:43:13.526276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.526246    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" event={"ID":"80241a2b-be47-424f-a5e3-4c96ef8e3802","Type":"ContainerStarted","Data":"4e6da093ba1fa861b94c6847204152c16f99129cf805a46764b8114e56610eb9"}
Nov 03 10:43:13.546865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.546802    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" podStartSLOduration=1.546784382 podStartE2EDuration="1.546784382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:13.544548307 +0000 UTC m=+823.750130359" watchObservedRunningTime="2026-11-03 10:43:13.546784382 +0000 UTC m=+823.752366440"
Nov 03 10:43:13.819655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.818668    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:13.850212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.850168    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 10:43:13.850376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:13.850243    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 10:43:14.041176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:14.041142    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:14.751293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:14.751037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:14.823593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:14.823218    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 10:43:14.823593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:14.823281    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 10:43:14.849953 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:14.849921    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 10:43:14.850183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:14.850158    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 10:43:15.525982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:15.524240    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:15.677377 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:15.676899    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:15.832662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:15.821924    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 10:43:15.832662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:15.821985    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 10:43:15.849883 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:15.849852    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 10:43:15.850127 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:15.850106    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 10:43:16.261350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.261318    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:16.819597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.818686    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 10:43:16.819597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.818745    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 10:43:16.851601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.850951    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 10:43:16.851601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.851011    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 10:43:16.871401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.871319    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"]
Nov 03 10:43:16.875435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.875415    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.886149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.886054    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"]
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935238    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-kube-api-access-8x7cp\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935307    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935337    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935402    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935435    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935472    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935545    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935622    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.939594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.935647    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:16.962595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:16.956201    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040256    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040338    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040366    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040470    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-kube-api-access-8x7cp\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040517    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040542    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040627    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040658    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.040698    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.045598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.041209    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.059725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.046978    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.059725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.049112    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.059725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.056246    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.059725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.056514    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.060904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.060875    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.061087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.061070    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.061178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.061161    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.062040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.062024    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-kube-api-access-8x7cp\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.212596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.207772    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:17.628790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.628753    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"]
Nov 03 10:43:17.632361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.632320    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:17.632475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.632412    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:17.632475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.632458    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:17.657770 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.657745    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:17.819597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.819009    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 10:43:17.819597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.819092    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 10:43:17.854873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.849972    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 10:43:17.854873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:17.850052    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 10:43:18.351434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:18.351402    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:18.578852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:18.578812    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" event={"ID":"e2b1448c-b3f3-4409-8cf0-0ef57081b431","Type":"ContainerStarted","Data":"8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a"}
Nov 03 10:43:18.579046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:18.578864    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" event={"ID":"e2b1448c-b3f3-4409-8cf0-0ef57081b431","Type":"ContainerStarted","Data":"44f6bd70bdf054f9f838e524eb846a579dd79c56982ab7fface192bec0422acc"}
Nov 03 10:43:18.612306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:18.605622    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" podStartSLOduration=2.6056037229999998 podStartE2EDuration="2.605603723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:18.597062155 +0000 UTC m=+828.802644207" watchObservedRunningTime="2026-11-03 10:43:18.605603723 +0000 UTC m=+828.811185776"
Nov 03 10:43:18.819073 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:18.819032    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 10:43:18.819381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:18.819351    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 10:43:18.850099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:18.850067    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 10:43:18.850435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:18.850375    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 10:43:19.145598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.144822    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:19.211629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.211600    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:19.212129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.212104    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:43:19.212317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.212291    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 10:43:19.653738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.653689    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-5m79x"]
Nov 03 10:43:19.659253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.659225    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.666945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.666335    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-5m79x"]
Nov 03 10:43:19.717332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.717293    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kx85\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-kube-api-access-2kx85\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.722642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.722609    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-plugins-config-volume\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.726605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.722949    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-provider-credentials\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.726845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.726823    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-routing-overlay-volume\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.741745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.741723    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-83bc832de857\""
Nov 03 10:43:19.821981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.821943    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 10:43:19.822258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.822226    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 10:43:19.828327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.828293    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kx85\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-kube-api-access-2kx85\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.829217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.829196    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-plugins-config-volume\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.830300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.830195    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-plugins-config-volume\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.830550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.830530    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-provider-credentials\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.830743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.830726    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-routing-overlay-volume\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.837594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.831519    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-routing-overlay-volume\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.843353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.843329    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kx85\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-kube-api-access-2kx85\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.849332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.849308    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-provider-credentials\") pod \"payload-processing-external-model-74b7555969-5m79x\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:19.994809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:19.994771    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:20.135602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.132110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:20.223481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.223068    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:43:20.223481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.223122    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 10:43:20.478753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.478599    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-5m79x"]
Nov 03 10:43:20.645909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.645875    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x" event={"ID":"ef7529ff-d218-4cc1-9b06-55aa0e9ed972","Type":"ContainerStarted","Data":"68741c0597a5592fa541dfeea1f2ce19b969fb44e41f6d6d5d0a8629e779ffc2"}
Nov 03 10:43:20.829650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.829214    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:20.830290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.830102    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:20.831132 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.831080    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:43:20.898458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.897859    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:43:20.898458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.897919    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:20.962265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:20.962223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:21.032639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.023563    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:21.032639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.025046    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:21.041634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.040971    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:43:21.113651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.113238    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:43:21.215598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.214944    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:43:21.215598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.215001    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 10:43:21.689596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.689361    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x" event={"ID":"ef7529ff-d218-4cc1-9b06-55aa0e9ed972","Type":"ContainerStarted","Data":"cbedc231864054e585acfc7986686f65335958bb56a3c185d2b83a41b993dd47"}
Nov 03 10:43:21.701598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.694154    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:21.711595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.707656    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x" podStartSLOduration=2.707639648 podStartE2EDuration="2.707639648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:21.707061758 +0000 UTC m=+831.912643813" watchObservedRunningTime="2026-11-03 10:43:21.707639648 +0000 UTC m=+831.913221696"
Nov 03 10:43:21.885554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:21.885517    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:22.210597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:22.209912    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:43:22.210597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:22.209969    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 10:43:22.463705 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:43:22.461791    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:43:22.655618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:22.652076    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:23.212596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:23.209908    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:43:23.212596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:23.209966    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 10:43:23.311598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:23.308173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:24.078300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:24.078257    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:24.209083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:24.209053    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:43:24.209259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:24.209105    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 10:43:25.278385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:25.278350    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:25.279507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:25.279480    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:25.280348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:25.280315    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:43:26.398043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:26.398011    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:26.981922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:26.981880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:27.609356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:27.609314    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:28.264766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:28.264729    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:28.943717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:28.943681    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:29.547102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:29.547069    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:30.262400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:30.261705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:30.868775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:30.868720    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:31.579492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.579452    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:31.892644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.892542    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"]
Nov 03 10:43:31.900283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.900251    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.911134 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.911105    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"]
Nov 03 10:43:31.953738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.953713    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh7fr\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-kube-api-access-nh7fr\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.953894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.953868    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.953990 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.953979    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.954036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.954011    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.954086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.954075    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.954121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.954111    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.954154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.954138    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.954275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.954260    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:31.954309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:31.954289    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065274    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065364    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065401    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065463    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh7fr\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-kube-api-access-nh7fr\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065550    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065631    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065672    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065719    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.065768    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.066865    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.067593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.067131    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.068283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.067818    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.074035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.068393    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.074035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.068979    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.095589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.084991    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.095589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.085504    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.095589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.087017    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.095589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.088586    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh7fr\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-kube-api-access-nh7fr\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.231595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.230941    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:32.614082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.613981    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:32.790786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.755168    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"]
Nov 03 10:43:32.807484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.807448    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:32.807633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.807583    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:32.807709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.807630    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:32.836093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:32.836064    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" event={"ID":"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1","Type":"ContainerStarted","Data":"d4500c18997d9004baadb887328645bb99eedbf641ddf021a9fabd3dfebd9d34"}
Nov 03 10:43:33.753772 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:33.753739    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:43:33.800699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:33.789851    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:33.963317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:33.963270    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"]
Nov 03 10:43:34.155985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:34.155891    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" event={"ID":"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1","Type":"ContainerStarted","Data":"1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83"}
Nov 03 10:43:34.276338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:34.276309    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:34.288317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:34.219083    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podStartSLOduration=3.219062407 podStartE2EDuration="3.219062407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:34.156438262 +0000 UTC m=+844.362020313" watchObservedRunningTime="2026-11-03 10:43:34.219062407 +0000 UTC m=+844.424644459"
Nov 03 10:43:34.288317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:34.277057    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:43:34.288317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:34.277104    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 10:43:34.319645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:34.318928    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:35.232991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:35.232957    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:43:35.233616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:35.233562    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 10:43:35.269302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:35.269268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:35.320700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:35.320658    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:35.321550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:35.321525    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:43:35.374719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:35.374675    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:36.202598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.201815    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:36.246284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.246248    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:43:36.246885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.246849    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 10:43:36.247411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.247386    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:36.305594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.299117    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:36.329598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.323664    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:43:36.378597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.377872    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:36.378597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.378488    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:43:36.501658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.487641    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:36.649731 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.649696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:36.749597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.747176    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:43:36.783158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:36.779520    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:43:37.134653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.109665    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-4s98l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:43:37.134653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.109738    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:37.233218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.233186    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:43:37.233458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.233434    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 10:43:37.398610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.397410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:37.847163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.847132    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"]
Nov 03 10:43:37.851376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.851352    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:37.857468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.857444    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"distinct-2-w2-e78c33-kserve-self-signed-certs\""
Nov 03 10:43:37.862371 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.862347    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"]
Nov 03 10:43:37.941420 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.941388    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:37.951199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.951176    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-home\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:37.951316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.951239    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tls-certs\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:37.951316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.951268    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgfm\" (UniqueName: \"kubernetes.io/projected/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-kube-api-access-rrgfm\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:37.951458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.951434    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tmp-dir\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:37.951635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.951613    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-dshm\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:37.951734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:37.951719    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-model-cache\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.057611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.054260    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-model-cache\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.057611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.054362    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-home\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.057611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.054427    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tls-certs\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.057611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.054451    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgfm\" (UniqueName: \"kubernetes.io/projected/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-kube-api-access-rrgfm\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.057611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.054511    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tmp-dir\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.057611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.054645    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-dshm\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.058530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.058503    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-model-cache\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.058736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.058557    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tmp-dir\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.058895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.058867    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-home\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.067878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.067853    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-dshm\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.068068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.068044    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tls-certs\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.070543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.070520    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgfm\" (UniqueName: \"kubernetes.io/projected/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-kube-api-access-rrgfm\") pod \"distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.168017 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.167938    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:38.234595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.233962    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:43:38.234595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.234012    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 10:43:38.343088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.343022    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:38.490047 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:43:38.490017    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod599a0d21_6f8c_433a_ab74_b9d0fc88c30a.slice/crio-5ea76b88affc9219064ed41ab7daf7f67fbbbc7be0915b5a01bd81cbfc47e8a1 WatchSource:0}: Error finding container 5ea76b88affc9219064ed41ab7daf7f67fbbbc7be0915b5a01bd81cbfc47e8a1: Status 404 returned error can't find the container with id 5ea76b88affc9219064ed41ab7daf7f67fbbbc7be0915b5a01bd81cbfc47e8a1
Nov 03 10:43:38.490497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.490297    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"]
Nov 03 10:43:39.002611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:38.999630    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8" podUID="7eef0755-9967-45d0-af8e-04bb96178702" containerName="payload-processing" containerID="cri-o://cc1b755701c703ec2c2542e51a13838ae716239773829f98fdf9972f02a3ebff" gracePeriod=25
Nov 03 10:43:39.078448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.078416    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:39.232293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.232252    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:43:39.232562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.232531    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 10:43:39.251914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.251880    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" event={"ID":"599a0d21-6f8c-433a-ab74-b9d0fc88c30a","Type":"ContainerStarted","Data":"027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc"}
Nov 03 10:43:39.252105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.251943    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" event={"ID":"599a0d21-6f8c-433a-ab74-b9d0fc88c30a","Type":"ContainerStarted","Data":"5ea76b88affc9219064ed41ab7daf7f67fbbbc7be0915b5a01bd81cbfc47e8a1"}
Nov 03 10:43:39.254060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.253971    5817 generic.go:356] "Generic (PLEG): container finished" podID="7eef0755-9967-45d0-af8e-04bb96178702" containerID="cc1b755701c703ec2c2542e51a13838ae716239773829f98fdf9972f02a3ebff" exitCode=0
Nov 03 10:43:39.254060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.253997    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8" event={"ID":"7eef0755-9967-45d0-af8e-04bb96178702","Type":"ContainerDied","Data":"cc1b755701c703ec2c2542e51a13838ae716239773829f98fdf9972f02a3ebff"}
Nov 03 10:43:39.270267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.270224    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podStartSLOduration=2.270212704 podStartE2EDuration="2.270212704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:39.267834319 +0000 UTC m=+849.473416370" watchObservedRunningTime="2026-11-03 10:43:39.270212704 +0000 UTC m=+849.475794755"
Nov 03 10:43:39.348177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.348154    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:39.371151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.371129    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-provider-credentials\") pod \"7eef0755-9967-45d0-af8e-04bb96178702\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") "
Nov 03 10:43:39.371261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.371213    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-kube-api-access-7hdmd\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-kube-api-access-7hdmd\") pod \"7eef0755-9967-45d0-af8e-04bb96178702\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") "
Nov 03 10:43:39.371309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.371291    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-routing-overlay-volume\") pod \"7eef0755-9967-45d0-af8e-04bb96178702\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") "
Nov 03 10:43:39.371477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.371456    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-plugins-config-volume\") pod \"7eef0755-9967-45d0-af8e-04bb96178702\" (UID: \"7eef0755-9967-45d0-af8e-04bb96178702\") "
Nov 03 10:43:39.371639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.371617    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-routing-overlay-volume" pod "7eef0755-9967-45d0-af8e-04bb96178702" (UID: "7eef0755-9967-45d0-af8e-04bb96178702"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:43:39.371935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.371890    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-plugins-config-volume" pod "7eef0755-9967-45d0-af8e-04bb96178702" (UID: "7eef0755-9967-45d0-af8e-04bb96178702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:43:39.372508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.372149    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:43:39.372508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.372172    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eef0755-9967-45d0-af8e-04bb96178702-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:43:39.374554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.374529    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-kube-api-access-7hdmd" pod "7eef0755-9967-45d0-af8e-04bb96178702" (UID: "7eef0755-9967-45d0-af8e-04bb96178702"). InnerVolumeSpecName "kube-api-access-7hdmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:43:39.374692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.374594    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-provider-credentials" pod "7eef0755-9967-45d0-af8e-04bb96178702" (UID: "7eef0755-9967-45d0-af8e-04bb96178702"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:43:39.473083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.473055    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:43:39.473083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:39.473081    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdmd\" (UniqueName: \"kubernetes.io/projected/7eef0755-9967-45d0-af8e-04bb96178702-kube-api-access-7hdmd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:43:40.048836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.048790    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:40.168611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.168556    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:40.232090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.232004    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:43:40.232090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.232066    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 10:43:40.262691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.262657    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"
Nov 03 10:43:40.262691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.262670    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8" event={"ID":"7eef0755-9967-45d0-af8e-04bb96178702","Type":"ContainerDied","Data":"4840daef7fe8d90a1b45de3c7e54b0d31331b4728e0efac2a0df3253ed9f5726"}
Nov 03 10:43:40.262911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.262731    5817 scope.go:122] "RemoveContainer" containerID="cc1b755701c703ec2c2542e51a13838ae716239773829f98fdf9972f02a3ebff"
Nov 03 10:43:40.281846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.281763    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"]
Nov 03 10:43:40.284931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.284909    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8bb6cf694-ttfc8"]
Nov 03 10:43:40.411601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.411128    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:43:40.847334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:40.847301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:41.232121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:41.232088    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:43:41.232621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:41.232147    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 10:43:41.308124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:41.308088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:41.738001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:41.737970    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:41.902495 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:41.902463    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eef0755-9967-45d0-af8e-04bb96178702" path="/var/lib/kubelet/pods/7eef0755-9967-45d0-af8e-04bb96178702/volumes"
Nov 03 10:43:42.237016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:42.236989    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:42.237467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:42.237403    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:42.238120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:42.238100    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 10:43:42.476240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:42.476205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:43.146841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:43.146809    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:43.883822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:43.883783    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:44.104680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.104648    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:44.277894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.277164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:44.455080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.455050    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5944845765-fcr4c"]
Nov 03 10:43:44.455605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.455591    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7eef0755-9967-45d0-af8e-04bb96178702" containerName="payload-processing"
Nov 03 10:43:44.455659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.455607    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eef0755-9967-45d0-af8e-04bb96178702" containerName="payload-processing"
Nov 03 10:43:44.455738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.455729    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eef0755-9967-45d0-af8e-04bb96178702" containerName="payload-processing"
Nov 03 10:43:44.459164 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.459147    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.466222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.466194    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-22130322f488\""
Nov 03 10:43:44.467381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.467360    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-89fdd827fb59\""
Nov 03 10:43:44.469872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.469852    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5944845765-fcr4c"]
Nov 03 10:43:44.561711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.561618    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-routing-overlay-volume\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.561861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.561780    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-provider-credentials\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.561935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.561916    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-plugins-config-volume\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.561977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.561954    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2854\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-kube-api-access-p2854\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.663072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.663032    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-plugins-config-volume\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.663237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.663082    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2854\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-kube-api-access-p2854\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.663237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.663170    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-routing-overlay-volume\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.663362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.663275    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-provider-credentials\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.663893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.663865    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-routing-overlay-volume\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.664036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.664019    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-plugins-config-volume\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.665908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.665883    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-provider-credentials\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.670634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.670613    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2854\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-kube-api-access-p2854\") pod \"payload-processing-external-model-5944845765-fcr4c\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") " pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.776656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.776630    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:44.862683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:44.860806    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:45.057623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.057587    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"]
Nov 03 10:43:45.062790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.062714    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.066065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.065868    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5944845765-fcr4c"]
Nov 03 10:43:45.066995 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:43:45.066968    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6db1dd_d8b2_4a95_bd52_4b44a7b611af.slice/crio-ee79099100995f964bb68f998156e96bdafc42c5bb7d9adb6274fe062621b632 WatchSource:0}: Error finding container ee79099100995f964bb68f998156e96bdafc42c5bb7d9adb6274fe062621b632: Status 404 returned error can't find the container with id ee79099100995f964bb68f998156e96bdafc42c5bb7d9adb6274fe062621b632
Nov 03 10:43:45.078273 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.078251    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"]
Nov 03 10:43:45.168768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.168739    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt2vq\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-kube-api-access-vt2vq\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.168898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.168789    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.168969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.168897    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d98ad-5d1e-4591-95f9-551ecedda34d-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.168969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.168950    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.169078 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.169014    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.169078 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.169044    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.169179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.169085    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.169248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.169198    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.169248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.169226    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270019    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270056    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270078    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt2vq\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-kube-api-access-vt2vq\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270126    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270189    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d98ad-5d1e-4591-95f9-551ecedda34d-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270237    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270285    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270326    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270392    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270435    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270518    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.270782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270765    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.271018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.270977    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d98ad-5d1e-4591-95f9-551ecedda34d-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.271171 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.271153    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.272517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.272499    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.273075 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.273059    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.277545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.277522    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.277765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.277749    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt2vq\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-kube-api-access-vt2vq\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.341010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.340978    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c" event={"ID":"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af","Type":"ContainerStarted","Data":"d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa"}
Nov 03 10:43:45.341169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.341037    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c" event={"ID":"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af","Type":"ContainerStarted","Data":"ee79099100995f964bb68f998156e96bdafc42c5bb7d9adb6274fe062621b632"}
Nov 03 10:43:45.341169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.341063    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:45.360088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.360003    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c" podStartSLOduration=1.35998794 podStartE2EDuration="1.35998794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:45.356063313 +0000 UTC m=+855.561645357" watchObservedRunningTime="2026-11-03 10:43:45.35998794 +0000 UTC m=+855.565569983"
Nov 03 10:43:45.407774 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.407747    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:45.672484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.672374    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:45.709077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.707945    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"]
Nov 03 10:43:45.709845 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:43:45.709456    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34d98ad_5d1e_4591_95f9_551ecedda34d.slice/crio-3236ae54a3adcbf8e8ac519856ee0160bed919555e1f1adb9d1b548eb37ffd65 WatchSource:0}: Error finding container 3236ae54a3adcbf8e8ac519856ee0160bed919555e1f1adb9d1b548eb37ffd65: Status 404 returned error can't find the container with id 3236ae54a3adcbf8e8ac519856ee0160bed919555e1f1adb9d1b548eb37ffd65
Nov 03 10:43:45.712350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.712320    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:45.712458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.712394    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:45.712458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:45.712437    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:46.356510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:46.356472    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" event={"ID":"e34d98ad-5d1e-4591-95f9-551ecedda34d","Type":"ContainerStarted","Data":"bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da"}
Nov 03 10:43:46.357040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:46.356559    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" event={"ID":"e34d98ad-5d1e-4591-95f9-551ecedda34d","Type":"ContainerStarted","Data":"3236ae54a3adcbf8e8ac519856ee0160bed919555e1f1adb9d1b548eb37ffd65"}
Nov 03 10:43:46.357604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:46.357546    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:46.378506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:46.378453    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podStartSLOduration=1.378437617 podStartE2EDuration="1.378437617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:46.374825996 +0000 UTC m=+856.580408053" watchObservedRunningTime="2026-11-03 10:43:46.378437617 +0000 UTC m=+856.584019827"
Nov 03 10:43:46.411600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:46.410911    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:46.418633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:46.414685    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:43:46.418633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:46.414745    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 10:43:46.613646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:46.613224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:47.220262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:47.220226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:47.411597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:47.410906    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:43:47.411597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:47.410958    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 10:43:48.109596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:48.109086    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:48.168473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:48.168432    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:48.184322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:48.184293    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:48.383484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:48.383407    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:43:48.410597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:48.409910    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:43:48.410597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:48.409963    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 10:43:49.071852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:49.071815    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:49.410068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:49.409979    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:43:49.410323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:49.410296    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 10:43:49.441797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:49.441767    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:50.217626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:50.212393    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:50.408876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:50.408839    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:43:50.409161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:50.409130    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 10:43:50.985455 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:50.985415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:51.412790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:51.412700    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:43:51.413393 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:51.413365    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 10:43:51.762607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:51.762531    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:52.416469 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:43:52.416414    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:43:52.466621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:52.463421    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:53.255274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:53.255238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:53.450748 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:53.450714    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:43:53.451241 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:53.450767    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:54.061400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:54.061373    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:54.061703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:54.061686    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:54.062448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:54.062424    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 10:43:54.123374 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:54.122941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:54.848204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:54.848171    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:54.976631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:54.976595    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:43:55.738538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:55.738507    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:56.369385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:56.369358    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:43:56.451600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:56.426471    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-5m79x"]
Nov 03 10:43:57.007673 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:57.007637    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:57.148478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:57.148445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:57.973261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:57.973171    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:59.047595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:59.047054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:43:59.216593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:59.201995    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:59.247794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:59.247500    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:59.774612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:43:59.773629    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:01.215697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.215664    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"]
Nov 03 10:44:01.221228 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.221177    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.226465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.226441    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:44:01.226628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.226495    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:44:01.227051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.227028    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"]
Nov 03 10:44:01.300782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.300753    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0361d514-284d-4774-bad5-65c9ce296669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.300972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.300825    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0361d514-284d-4774-bad5-65c9ce296669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.301031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.300976    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8mh\" (UniqueName: \"kubernetes.io/projected/0361d514-284d-4774-bad5-65c9ce296669-kube-api-access-pc8mh\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.402522 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.402493    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0361d514-284d-4774-bad5-65c9ce296669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.402743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.402564    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8mh\" (UniqueName: \"kubernetes.io/projected/0361d514-284d-4774-bad5-65c9ce296669-kube-api-access-pc8mh\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.402743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.402705    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0361d514-284d-4774-bad5-65c9ce296669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.403391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.403329    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0361d514-284d-4774-bad5-65c9ce296669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.405005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.404975    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0361d514-284d-4774-bad5-65c9ce296669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.413362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.413339    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8mh\" (UniqueName: \"kubernetes.io/projected/0361d514-284d-4774-bad5-65c9ce296669-kube-api-access-pc8mh\") pod \"maas-api-revoke-sub-5a3be1f5-xnzbw\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.426961 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.426934    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x" podUID="ef7529ff-d218-4cc1-9b06-55aa0e9ed972" containerName="payload-processing" containerID="cri-o://cbedc231864054e585acfc7986686f65335958bb56a3c185d2b83a41b993dd47" gracePeriod=25
Nov 03 10:44:01.505104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.505022    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:01.540642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.540608    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:01.642154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.640909    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:01.675738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.675706    5817 generic.go:356] "Generic (PLEG): container finished" podID="ef7529ff-d218-4cc1-9b06-55aa0e9ed972" containerID="cbedc231864054e585acfc7986686f65335958bb56a3c185d2b83a41b993dd47" exitCode=0
Nov 03 10:44:01.675886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.675778    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x" event={"ID":"ef7529ff-d218-4cc1-9b06-55aa0e9ed972","Type":"ContainerDied","Data":"cbedc231864054e585acfc7986686f65335958bb56a3c185d2b83a41b993dd47"}
Nov 03 10:44:01.782271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.782247    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:44:01.807426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.807401    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-provider-credentials\") pod \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") "
Nov 03 10:44:01.807602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.807441    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-plugins-config-volume\") pod \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") "
Nov 03 10:44:01.807602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.807473    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-routing-overlay-volume\") pod \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") "
Nov 03 10:44:01.807728 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.807629    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-kube-api-access-2kx85\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-kube-api-access-2kx85\") pod \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\" (UID: \"ef7529ff-d218-4cc1-9b06-55aa0e9ed972\") "
Nov 03 10:44:01.807900 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.807877    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-plugins-config-volume" pod "ef7529ff-d218-4cc1-9b06-55aa0e9ed972" (UID: "ef7529ff-d218-4cc1-9b06-55aa0e9ed972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:01.808157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.808132    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:01.808473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.808446    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-routing-overlay-volume" pod "ef7529ff-d218-4cc1-9b06-55aa0e9ed972" (UID: "ef7529ff-d218-4cc1-9b06-55aa0e9ed972"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:01.810226 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.810194    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-kube-api-access-2kx85" pod "ef7529ff-d218-4cc1-9b06-55aa0e9ed972" (UID: "ef7529ff-d218-4cc1-9b06-55aa0e9ed972"). InnerVolumeSpecName "kube-api-access-2kx85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:01.810330 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.810218    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-provider-credentials" pod "ef7529ff-d218-4cc1-9b06-55aa0e9ed972" (UID: "ef7529ff-d218-4cc1-9b06-55aa0e9ed972"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:01.871252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.871231    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"]
Nov 03 10:44:01.871677 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:01.871657    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0361d514_284d_4774_bad5_65c9ce296669.slice/crio-b4df624c62ad5c664f83b2f3a8c2776b652a7fcde1fdfb50e22e62d341f37309 WatchSource:0}: Error finding container b4df624c62ad5c664f83b2f3a8c2776b652a7fcde1fdfb50e22e62d341f37309: Status 404 returned error can't find the container with id b4df624c62ad5c664f83b2f3a8c2776b652a7fcde1fdfb50e22e62d341f37309
Nov 03 10:44:01.916998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.916964    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kx85\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-kube-api-access-2kx85\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:01.916998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.916996    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:01.917146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:01.917011    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7529ff-d218-4cc1-9b06-55aa0e9ed972-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:02.110683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.110644    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:02.685169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.685145    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x"
Nov 03 10:44:02.685750 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.685146    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74b7555969-5m79x" event={"ID":"ef7529ff-d218-4cc1-9b06-55aa0e9ed972","Type":"ContainerDied","Data":"68741c0597a5592fa541dfeea1f2ce19b969fb44e41f6d6d5d0a8629e779ffc2"}
Nov 03 10:44:02.685750 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.685304    5817 scope.go:122] "RemoveContainer" containerID="cbedc231864054e585acfc7986686f65335958bb56a3c185d2b83a41b993dd47"
Nov 03 10:44:02.687379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.687215    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw" event={"ID":"0361d514-284d-4774-bad5-65c9ce296669","Type":"ContainerStarted","Data":"257e5dc92d1d86dd6b04ce82f61b11a14189df7956cb79183c0a50d06e564fd2"}
Nov 03 10:44:02.687379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.687342    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw" event={"ID":"0361d514-284d-4774-bad5-65c9ce296669","Type":"ContainerStarted","Data":"b4df624c62ad5c664f83b2f3a8c2776b652a7fcde1fdfb50e22e62d341f37309"}
Nov 03 10:44:02.706819 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.706366    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw" podStartSLOduration=1.706350273 podStartE2EDuration="1.706350273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:02.70573261 +0000 UTC m=+872.911314661" watchObservedRunningTime="2026-11-03 10:44:02.706350273 +0000 UTC m=+872.911932325"
Nov 03 10:44:02.723288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.723264    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-5m79x"]
Nov 03 10:44:02.727876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.727855    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-5m79x"]
Nov 03 10:44:02.752364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:02.752329    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:03.348611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.347547    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:03.572013 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.571967    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"]
Nov 03 10:44:03.572966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.572940    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ef7529ff-d218-4cc1-9b06-55aa0e9ed972" containerName="payload-processing"
Nov 03 10:44:03.573152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.573128    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7529ff-d218-4cc1-9b06-55aa0e9ed972" containerName="payload-processing"
Nov 03 10:44:03.573373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.573358    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7529ff-d218-4cc1-9b06-55aa0e9ed972" containerName="payload-processing"
Nov 03 10:44:03.584017 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.583991    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"]
Nov 03 10:44:03.584169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.584105    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.588318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.588292    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"distinct-w4-d5d3c3-kserve-self-signed-certs\""
Nov 03 10:44:03.635564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.635477    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tmp-dir\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.635564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.635519    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tls-certs\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.635788 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.635631    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-dshm\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.635788 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.635690    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96dr\" (UniqueName: \"kubernetes.io/projected/4bc83089-3cd2-4b50-94c7-fc527102cbaa-kube-api-access-h96dr\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.635883 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.635779    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-model-cache\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.635953 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.635937    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-home\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.660065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.660033    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:03.701915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.701882    5817 generic.go:356] "Generic (PLEG): container finished" podID="0361d514-284d-4774-bad5-65c9ce296669" containerID="257e5dc92d1d86dd6b04ce82f61b11a14189df7956cb79183c0a50d06e564fd2" exitCode=0
Nov 03 10:44:03.701915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.702056    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw" event={"ID":"0361d514-284d-4774-bad5-65c9ce296669","Type":"ContainerDied","Data":"257e5dc92d1d86dd6b04ce82f61b11a14189df7956cb79183c0a50d06e564fd2"}
Nov 03 10:44:03.736824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.736781    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-home\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.737043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.736841    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tmp-dir\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.737043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.736868    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tls-certs\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.737043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.736948    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-dshm\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.737043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.737022    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96dr\" (UniqueName: \"kubernetes.io/projected/4bc83089-3cd2-4b50-94c7-fc527102cbaa-kube-api-access-h96dr\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.737261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.737058    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-model-cache\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.737630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.737561    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-model-cache\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.738756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.737839    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tmp-dir\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.738756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.738401    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-home\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.743605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.743542    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-dshm\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.753548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.751563    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96dr\" (UniqueName: \"kubernetes.io/projected/4bc83089-3cd2-4b50-94c7-fc527102cbaa-kube-api-access-h96dr\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.758902 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.758876    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tls-certs\") pod \"distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.900700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.900628    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:03.901851 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:03.901823    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7529ff-d218-4cc1-9b06-55aa0e9ed972" path="/var/lib/kubelet/pods/ef7529ff-d218-4cc1-9b06-55aa0e9ed972/volumes"
Nov 03 10:44:04.167511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.167432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:04.181066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.181037    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"]
Nov 03 10:44:04.182127 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:04.182101    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bc83089_3cd2_4b50_94c7_fc527102cbaa.slice/crio-8039d15120499e98f782a0814e7a696152e1747b7bbe151177793b6fe59a8036 WatchSource:0}: Error finding container 8039d15120499e98f782a0814e7a696152e1747b7bbe151177793b6fe59a8036: Status 404 returned error can't find the container with id 8039d15120499e98f782a0814e7a696152e1747b7bbe151177793b6fe59a8036
Nov 03 10:44:04.711424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.711380    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" event={"ID":"4bc83089-3cd2-4b50-94c7-fc527102cbaa","Type":"ContainerStarted","Data":"13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799"}
Nov 03 10:44:04.711424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.711428    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" event={"ID":"4bc83089-3cd2-4b50-94c7-fc527102cbaa","Type":"ContainerStarted","Data":"8039d15120499e98f782a0814e7a696152e1747b7bbe151177793b6fe59a8036"}
Nov 03 10:44:04.729095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.729036    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podStartSLOduration=1.7290176769999999 podStartE2EDuration="1.729017677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:04.726715556 +0000 UTC m=+874.932297622" watchObservedRunningTime="2026-11-03 10:44:04.729017677 +0000 UTC m=+874.934599728"
Nov 03 10:44:04.888854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.888716    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:04.892509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.892468    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:04.901997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.901965    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:04.959850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.959768    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0361d514-284d-4774-bad5-65c9ce296669-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0361d514-284d-4774-bad5-65c9ce296669-maas-api-openshift-ca\") pod \"0361d514-284d-4774-bad5-65c9ce296669\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") "
Nov 03 10:44:04.960038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.959904    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0361d514-284d-4774-bad5-65c9ce296669-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0361d514-284d-4774-bad5-65c9ce296669-maas-api-xks-ca\") pod \"0361d514-284d-4774-bad5-65c9ce296669\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") "
Nov 03 10:44:04.960131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.960068    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0361d514-284d-4774-bad5-65c9ce296669-kube-api-access-pc8mh\" (UniqueName: \"kubernetes.io/projected/0361d514-284d-4774-bad5-65c9ce296669-kube-api-access-pc8mh\") pod \"0361d514-284d-4774-bad5-65c9ce296669\" (UID: \"0361d514-284d-4774-bad5-65c9ce296669\") "
Nov 03 10:44:04.962251 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.961258    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0361d514-284d-4774-bad5-65c9ce296669-maas-api-openshift-ca" pod "0361d514-284d-4774-bad5-65c9ce296669" (UID: "0361d514-284d-4774-bad5-65c9ce296669"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:04.963994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.963949    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0361d514-284d-4774-bad5-65c9ce296669-maas-api-xks-ca" pod "0361d514-284d-4774-bad5-65c9ce296669" (UID: "0361d514-284d-4774-bad5-65c9ce296669"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:04.964631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:04.964603    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0361d514-284d-4774-bad5-65c9ce296669-kube-api-access-pc8mh" pod "0361d514-284d-4774-bad5-65c9ce296669" (UID: "0361d514-284d-4774-bad5-65c9ce296669"). InnerVolumeSpecName "kube-api-access-pc8mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:05.061920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.061885    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc8mh\" (UniqueName: \"kubernetes.io/projected/0361d514-284d-4774-bad5-65c9ce296669-kube-api-access-pc8mh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:05.061920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.061919    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0361d514-284d-4774-bad5-65c9ce296669-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:05.061920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.061935    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0361d514-284d-4774-bad5-65c9ce296669-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:05.550319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.550286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:05.587939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.587902    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"]
Nov 03 10:44:05.588533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.588514    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0361d514-284d-4774-bad5-65c9ce296669" containerName="revoke-keys"
Nov 03 10:44:05.588696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.588536    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0361d514-284d-4774-bad5-65c9ce296669" containerName="revoke-keys"
Nov 03 10:44:05.588764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.588709    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0361d514-284d-4774-bad5-65c9ce296669" containerName="revoke-keys"
Nov 03 10:44:05.601812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.601780    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"]
Nov 03 10:44:05.601980 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.601912    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.668981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.668957    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.669135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.668988    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vfbn\" (UniqueName: \"kubernetes.io/projected/3fefcd06-9c73-4dcc-80eb-af699f0fb418-kube-api-access-7vfbn\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.669135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.669018    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.720811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.720785    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"
Nov 03 10:44:05.721271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.720805    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw" event={"ID":"0361d514-284d-4774-bad5-65c9ce296669","Type":"ContainerDied","Data":"b4df624c62ad5c664f83b2f3a8c2776b652a7fcde1fdfb50e22e62d341f37309"}
Nov 03 10:44:05.721271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.720853    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4df624c62ad5c664f83b2f3a8c2776b652a7fcde1fdfb50e22e62d341f37309"
Nov 03 10:44:05.770137 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.770110    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.770286 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.770149    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vfbn\" (UniqueName: \"kubernetes.io/projected/3fefcd06-9c73-4dcc-80eb-af699f0fb418-kube-api-access-7vfbn\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.770286 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.770182    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.770812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.770789    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.772655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.772628    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.777981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.777960    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vfbn\" (UniqueName: \"kubernetes.io/projected/3fefcd06-9c73-4dcc-80eb-af699f0fb418-kube-api-access-7vfbn\") pod \"maas-api-revoke-sub-3b03480b-4ml2k\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:05.921129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:05.921057    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:06.201916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:06.201888    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"]
Nov 03 10:44:06.203874 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:06.203843    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fefcd06_9c73_4dcc_80eb_af699f0fb418.slice/crio-2d0a5c803ae2c917097ea58ff1539bd6ad9260e9ceb5d7edff9b25cdd1b380d5 WatchSource:0}: Error finding container 2d0a5c803ae2c917097ea58ff1539bd6ad9260e9ceb5d7edff9b25cdd1b380d5: Status 404 returned error can't find the container with id 2d0a5c803ae2c917097ea58ff1539bd6ad9260e9ceb5d7edff9b25cdd1b380d5
Nov 03 10:44:06.265402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:06.265369    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:06.736612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:06.731823    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k" event={"ID":"3fefcd06-9c73-4dcc-80eb-af699f0fb418","Type":"ContainerStarted","Data":"9a2fdca66fdd68f97f10e7f10f19707c07eb78b4e9e1c2b148f74312408f6316"}
Nov 03 10:44:06.736612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:06.731884    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k" event={"ID":"3fefcd06-9c73-4dcc-80eb-af699f0fb418","Type":"ContainerStarted","Data":"2d0a5c803ae2c917097ea58ff1539bd6ad9260e9ceb5d7edff9b25cdd1b380d5"}
Nov 03 10:44:06.751620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:06.751533    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k" podStartSLOduration=1.751521176 podStartE2EDuration="1.751521176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:06.748237328 +0000 UTC m=+876.953819391" watchObservedRunningTime="2026-11-03 10:44:06.751521176 +0000 UTC m=+876.957103220"
Nov 03 10:44:07.057899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:07.057856    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:07.752717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:07.752680    5817 generic.go:356] "Generic (PLEG): container finished" podID="3fefcd06-9c73-4dcc-80eb-af699f0fb418" containerID="9a2fdca66fdd68f97f10e7f10f19707c07eb78b4e9e1c2b148f74312408f6316" exitCode=0
Nov 03 10:44:07.753132 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:07.752779    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k" event={"ID":"3fefcd06-9c73-4dcc-80eb-af699f0fb418","Type":"ContainerDied","Data":"9a2fdca66fdd68f97f10e7f10f19707c07eb78b4e9e1c2b148f74312408f6316"}
Nov 03 10:44:07.840980 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:07.840941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:08.301219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.301187    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"]
Nov 03 10:44:08.306139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.306116    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.314207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.314181    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"]
Nov 03 10:44:08.314564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.314544    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"openshift-service-ca.crt\""
Nov 03 10:44:08.315999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.315971    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"kube-root-ca.crt\""
Nov 03 10:44:08.316097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.316040    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"distinct-w3-b1c6f6-kserve-self-signed-certs\""
Nov 03 10:44:08.316232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.316217    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"default-dockercfg-lpsf5\""
Nov 03 10:44:08.380249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.380218    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:08.407564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.407538    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-tmp-dir\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.407700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.407675    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5cf\" (UniqueName: \"kubernetes.io/projected/29455243-b7f4-4059-8066-ddf8e33d6abd-kube-api-access-bt5cf\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.407741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.407718    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-dshm\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.407782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.407767    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-model-cache\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.407887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.407871    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-home\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.408054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.408032    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29455243-b7f4-4059-8066-ddf8e33d6abd-tls-certs\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.509891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.509849    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29455243-b7f4-4059-8066-ddf8e33d6abd-tls-certs\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.510151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.509913    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-tmp-dir\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.510151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.510002    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5cf\" (UniqueName: \"kubernetes.io/projected/29455243-b7f4-4059-8066-ddf8e33d6abd-kube-api-access-bt5cf\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.510151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.510030    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-dshm\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.510151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.510075    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-model-cache\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.510151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.510127    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-home\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.510647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.510623    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-home\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.511799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.511778    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-model-cache\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.512208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.512190    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-tmp-dir\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.513885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.513861    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29455243-b7f4-4059-8066-ddf8e33d6abd-tls-certs\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.514071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.514053    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-dshm\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.519760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.519727    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5cf\" (UniqueName: \"kubernetes.io/projected/29455243-b7f4-4059-8066-ddf8e33d6abd-kube-api-access-bt5cf\") pod \"distinct-w3-b1c6f6-kserve-69d9568886-84nlf\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.632480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.632392    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:08.898356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.898275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:08.918530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.918438    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"]
Nov 03 10:44:08.919521 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:08.919499    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29455243_b7f4_4059_8066_ddf8e33d6abd.slice/crio-7e839854773b5005691e1b5b3a3865ac44389ba23601d4f68cccd195ae4dde26 WatchSource:0}: Error finding container 7e839854773b5005691e1b5b3a3865ac44389ba23601d4f68cccd195ae4dde26: Status 404 returned error can't find the container with id 7e839854773b5005691e1b5b3a3865ac44389ba23601d4f68cccd195ae4dde26
Nov 03 10:44:08.928672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:08.928652    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:09.018438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.018402    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-openshift-ca\") pod \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") "
Nov 03 10:44:09.018438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.018485    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-xks-ca\") pod \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") "
Nov 03 10:44:09.018731 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.018607    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fefcd06-9c73-4dcc-80eb-af699f0fb418-kube-api-access-7vfbn\" (UniqueName: \"kubernetes.io/projected/3fefcd06-9c73-4dcc-80eb-af699f0fb418-kube-api-access-7vfbn\") pod \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\" (UID: \"3fefcd06-9c73-4dcc-80eb-af699f0fb418\") "
Nov 03 10:44:09.019181 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.019008    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-openshift-ca" pod "3fefcd06-9c73-4dcc-80eb-af699f0fb418" (UID: "3fefcd06-9c73-4dcc-80eb-af699f0fb418"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:09.019290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.019205    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:09.021866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.021829    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-xks-ca" pod "3fefcd06-9c73-4dcc-80eb-af699f0fb418" (UID: "3fefcd06-9c73-4dcc-80eb-af699f0fb418"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:09.021866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.021857    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fefcd06-9c73-4dcc-80eb-af699f0fb418-kube-api-access-7vfbn" pod "3fefcd06-9c73-4dcc-80eb-af699f0fb418" (UID: "3fefcd06-9c73-4dcc-80eb-af699f0fb418"). InnerVolumeSpecName "kube-api-access-7vfbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:09.120263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.120234    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fefcd06-9c73-4dcc-80eb-af699f0fb418-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:09.120263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.120261    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vfbn\" (UniqueName: \"kubernetes.io/projected/3fefcd06-9c73-4dcc-80eb-af699f0fb418-kube-api-access-7vfbn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:09.670014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.669977    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:09.768957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.768931    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"
Nov 03 10:44:09.769141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.768957    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k" event={"ID":"3fefcd06-9c73-4dcc-80eb-af699f0fb418","Type":"ContainerDied","Data":"2d0a5c803ae2c917097ea58ff1539bd6ad9260e9ceb5d7edff9b25cdd1b380d5"}
Nov 03 10:44:09.769141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.769011    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d0a5c803ae2c917097ea58ff1539bd6ad9260e9ceb5d7edff9b25cdd1b380d5"
Nov 03 10:44:09.770643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.770618    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" event={"ID":"29455243-b7f4-4059-8066-ddf8e33d6abd","Type":"ContainerStarted","Data":"9679a69ef07a09ee16b0884e36dbbd3c4ed8968703d0cc677742354244738756"}
Nov 03 10:44:09.770771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.770651    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" event={"ID":"29455243-b7f4-4059-8066-ddf8e33d6abd","Type":"ContainerStarted","Data":"7e839854773b5005691e1b5b3a3865ac44389ba23601d4f68cccd195ae4dde26"}
Nov 03 10:44:09.794828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:09.794781    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podStartSLOduration=1.7947656269999999 podStartE2EDuration="1.794765627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:09.790653222 +0000 UTC m=+879.996235274" watchObservedRunningTime="2026-11-03 10:44:09.794765627 +0000 UTC m=+880.000347679"
Nov 03 10:44:10.635597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:10.634659    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:10.743763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:10.743719    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:12.251112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:12.251079    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:13.061500 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:13.061470    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:13.687261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:13.687225    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:13.910271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:13.910240    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:13.919186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:13.919160    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:14.173537 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:14.173454    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:14.505628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:14.500447    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:14.879853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:14.879775    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:15.175398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.175249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:15.948455 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.948417    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:15.965759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.965724    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"]
Nov 03 10:44:15.966332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.966314    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3fefcd06-9c73-4dcc-80eb-af699f0fb418" containerName="revoke-keys"
Nov 03 10:44:15.966431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.966334    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fefcd06-9c73-4dcc-80eb-af699f0fb418" containerName="revoke-keys"
Nov 03 10:44:15.966501 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.966490    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fefcd06-9c73-4dcc-80eb-af699f0fb418" containerName="revoke-keys"
Nov 03 10:44:15.969979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.969960    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:15.976334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.976315    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:44:15.976466 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.976352    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:44:15.978665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:15.978643    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"]
Nov 03 10:44:16.105972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.105943    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qpfz\" (UniqueName: \"kubernetes.io/projected/a772b33f-fc26-4016-a727-07adf1fd3e5d-kube-api-access-2qpfz\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.106170 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.106147    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.106269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.106255    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.207470 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.207396    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.207470 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.207461    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.207738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.207495    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qpfz\" (UniqueName: \"kubernetes.io/projected/a772b33f-fc26-4016-a727-07adf1fd3e5d-kube-api-access-2qpfz\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.208095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.208069    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.209935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.209914    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.215343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.215320    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qpfz\" (UniqueName: \"kubernetes.io/projected/a772b33f-fc26-4016-a727-07adf1fd3e5d-kube-api-access-2qpfz\") pod \"maas-api-revoke-sub-9dd8b1cb-tvlzd\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.287324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.287294    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:16.570278 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:16.570250    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda772b33f_fc26_4016_a727_07adf1fd3e5d.slice/crio-908b8d32c460edafb13631cb8be927908df8b2ac15134864511f916bb928d599 WatchSource:0}: Error finding container 908b8d32c460edafb13631cb8be927908df8b2ac15134864511f916bb928d599: Status 404 returned error can't find the container with id 908b8d32c460edafb13631cb8be927908df8b2ac15134864511f916bb928d599
Nov 03 10:44:16.571056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.571023    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"]
Nov 03 10:44:16.661837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.661801    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:16.777108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.777070    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-r8628"]
Nov 03 10:44:16.783354 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.783329    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.819644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.789449    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-r8628"]
Nov 03 10:44:16.844629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.843998    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:16.876595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.875655    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-routing-overlay-volume\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.876595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.875773    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-plugins-config-volume\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.876595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.875816    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-provider-credentials\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.876595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.875960    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgqq\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-kube-api-access-6hgqq\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.976597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.971718    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd" event={"ID":"a772b33f-fc26-4016-a727-07adf1fd3e5d","Type":"ContainerStarted","Data":"908b8d32c460edafb13631cb8be927908df8b2ac15134864511f916bb928d599"}
Nov 03 10:44:16.985565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.977792    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-provider-credentials\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.985565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.978001    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgqq\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-kube-api-access-6hgqq\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.985565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.978141    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-routing-overlay-volume\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.985565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.978232    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-plugins-config-volume\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.985565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.979398    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-plugins-config-volume\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.985565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.982643    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-provider-credentials\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.985565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.984179    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-routing-overlay-volume\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:16.998927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:16.998868    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgqq\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-kube-api-access-6hgqq\") pod \"payload-processing-external-model-74b7555969-r8628\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") " pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:17.207193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.207164    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:17.503246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.503175    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-r8628"]
Nov 03 10:44:17.506751 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:17.506722    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14bd8eb_44df_4d3b_b2a7_b4b8de9fc022.slice/crio-009382091cb197009812b50baef57827c9a6e89021460c43717af056197181fc WatchSource:0}: Error finding container 009382091cb197009812b50baef57827c9a6e89021460c43717af056197181fc: Status 404 returned error can't find the container with id 009382091cb197009812b50baef57827c9a6e89021460c43717af056197181fc
Nov 03 10:44:17.653877 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.653751    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:17.851667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.851550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:17.980708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.980673    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628" event={"ID":"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022","Type":"ContainerStarted","Data":"9ad480e29171c66e4ce89d13a59783f93b410755f9453de7e913d6eba09c4c42"}
Nov 03 10:44:17.981149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.980730    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628" event={"ID":"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022","Type":"ContainerStarted","Data":"009382091cb197009812b50baef57827c9a6e89021460c43717af056197181fc"}
Nov 03 10:44:17.981149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.980777    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:17.982515 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.982491    5817 generic.go:356] "Generic (PLEG): container finished" podID="a772b33f-fc26-4016-a727-07adf1fd3e5d" containerID="b958985731dd364fb29153b73715383469c6f493b6174dd94071e5bb9226f484" exitCode=0
Nov 03 10:44:17.982662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.982562    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd" event={"ID":"a772b33f-fc26-4016-a727-07adf1fd3e5d","Type":"ContainerDied","Data":"b958985731dd364fb29153b73715383469c6f493b6174dd94071e5bb9226f484"}
Nov 03 10:44:18.000014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:17.999965    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628" podStartSLOduration=1.9999493940000002 podStartE2EDuration="1.999949394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:17.996348445 +0000 UTC m=+888.201930496" watchObservedRunningTime="2026-11-03 10:44:17.999949394 +0000 UTC m=+888.205531448"
Nov 03 10:44:18.447252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:18.447179    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:18.633444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:18.633401    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:18.646060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:18.646033    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:19.005370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.005341    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:19.156297 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.156273    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:19.215531 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.215500    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a772b33f-fc26-4016-a727-07adf1fd3e5d-kube-api-access-2qpfz\" (UniqueName: \"kubernetes.io/projected/a772b33f-fc26-4016-a727-07adf1fd3e5d-kube-api-access-2qpfz\") pod \"a772b33f-fc26-4016-a727-07adf1fd3e5d\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") "
Nov 03 10:44:19.215716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.215681    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-openshift-ca\") pod \"a772b33f-fc26-4016-a727-07adf1fd3e5d\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") "
Nov 03 10:44:19.215784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.215762    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-xks-ca\") pod \"a772b33f-fc26-4016-a727-07adf1fd3e5d\" (UID: \"a772b33f-fc26-4016-a727-07adf1fd3e5d\") "
Nov 03 10:44:19.216193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.216152    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-openshift-ca" pod "a772b33f-fc26-4016-a727-07adf1fd3e5d" (UID: "a772b33f-fc26-4016-a727-07adf1fd3e5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:19.216394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.216375    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:19.218257 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.218225    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a772b33f-fc26-4016-a727-07adf1fd3e5d-kube-api-access-2qpfz" pod "a772b33f-fc26-4016-a727-07adf1fd3e5d" (UID: "a772b33f-fc26-4016-a727-07adf1fd3e5d"). InnerVolumeSpecName "kube-api-access-2qpfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:19.218365 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.218296    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-xks-ca" pod "a772b33f-fc26-4016-a727-07adf1fd3e5d" (UID: "a772b33f-fc26-4016-a727-07adf1fd3e5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:19.271591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.266529    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:19.317758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.317711    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a772b33f-fc26-4016-a727-07adf1fd3e5d-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:19.317758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:19.317753    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qpfz\" (UniqueName: \"kubernetes.io/projected/a772b33f-fc26-4016-a727-07adf1fd3e5d-kube-api-access-2qpfz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:20.004108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:20.004078    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"
Nov 03 10:44:20.004287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:20.004083    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd" event={"ID":"a772b33f-fc26-4016-a727-07adf1fd3e5d","Type":"ContainerDied","Data":"908b8d32c460edafb13631cb8be927908df8b2ac15134864511f916bb928d599"}
Nov 03 10:44:20.004287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:20.004175    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="908b8d32c460edafb13631cb8be927908df8b2ac15134864511f916bb928d599"
Nov 03 10:44:20.044911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:20.044658    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:20.852188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:20.852145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:21.459857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:21.459818    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:22.013634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:22.013597    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:44:22.170433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:22.170397    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:22.412460 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:44:22.412380    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:44:23.147054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:23.147022    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:25.130598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:25.123716    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:44:25.130598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:25.123787    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:25.609986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:25.609949    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:26.397497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:26.397464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:26.534325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:26.534296    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:27.223764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.223711    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"]
Nov 03 10:44:27.224553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.224530    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a772b33f-fc26-4016-a727-07adf1fd3e5d" containerName="revoke-keys"
Nov 03 10:44:27.224553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.224553    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a772b33f-fc26-4016-a727-07adf1fd3e5d" containerName="revoke-keys"
Nov 03 10:44:27.224779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.224765    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a772b33f-fc26-4016-a727-07adf1fd3e5d" containerName="revoke-keys"
Nov 03 10:44:27.229389 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.229368    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.240171 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.240144    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"]
Nov 03 10:44:27.338180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338143    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vn5\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-kube-api-access-r2vn5\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.338328 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338189    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.338328 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338254    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.338328 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338279    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.338328 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338312    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.338328 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338328    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.338566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338362    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.338566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338443    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.338691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.338563    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b2d21f-effb-4929-8c92-c4d49af1499e-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.391196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.391148    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:27.439476 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439437    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b2d21f-effb-4929-8c92-c4d49af1499e-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.439965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439543    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vn5\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-kube-api-access-r2vn5\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.439965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439592    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.439965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439627    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.439965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439643    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.439965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439669    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.439965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439695    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.439965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439711    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.439965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439758    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.440299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.439997    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.440299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.440144    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.440299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.440240    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.440299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.440277    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.440299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.440285    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b2d21f-effb-4929-8c92-c4d49af1499e-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.442429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.442401    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.442872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.442845    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.447945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.447916    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.449749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.449723    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vn5\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-kube-api-access-r2vn5\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.555053 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.554989    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:27.881023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.880993    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"]
Nov 03 10:44:27.882319 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:27.882294    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b2d21f_effb_4929_8c92_c4d49af1499e.slice/crio-8415c2fc7f1e0c729e54f29647cd60dfbdbc8b5024cc7bbe96decaeb861d5364 WatchSource:0}: Error finding container 8415c2fc7f1e0c729e54f29647cd60dfbdbc8b5024cc7bbe96decaeb861d5364: Status 404 returned error can't find the container with id 8415c2fc7f1e0c729e54f29647cd60dfbdbc8b5024cc7bbe96decaeb861d5364
Nov 03 10:44:27.885024 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.884996    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:27.885102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.885058    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:27.885102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:27.885087    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:28.176383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.176300    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" event={"ID":"68b2d21f-effb-4929-8c92-c4d49af1499e","Type":"ContainerStarted","Data":"fe98a4cd90dcfdcb7f742271acbb44ce3e67139f2de810783a1f660a51fb2e06"}
Nov 03 10:44:28.176383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.176365    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" event={"ID":"68b2d21f-effb-4929-8c92-c4d49af1499e","Type":"ContainerStarted","Data":"8415c2fc7f1e0c729e54f29647cd60dfbdbc8b5024cc7bbe96decaeb861d5364"}
Nov 03 10:44:28.196383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.196330    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podStartSLOduration=1.196315371 podStartE2EDuration="1.196315371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:28.195242402 +0000 UTC m=+898.400824479" watchObservedRunningTime="2026-11-03 10:44:28.196315371 +0000 UTC m=+898.401897422"
Nov 03 10:44:28.270390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.270302    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:28.555959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.555931    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:28.557856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.557828    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 10:44:28.557996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.557897    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 10:44:28.766180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.766145    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-r8628"]
Nov 03 10:44:28.769301 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.769282    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:28.772584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.772540    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5944845765-fcr4c"]
Nov 03 10:44:28.836635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.836495    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:28.980659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:28.977182    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:29.339643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:29.339604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:29.555763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:29.555723    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 10:44:29.555960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:29.555782    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 10:44:30.120184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.120151    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"]
Nov 03 10:44:30.134207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.134173    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"]
Nov 03 10:44:30.134344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.134219    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"]
Nov 03 10:44:30.134473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.134459    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.138916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.138895    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.153304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.153275    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"]
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.186848    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7818f918-9435-469c-a3b6-c16e509f15ea-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.186913    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81954e6-e83d-44b4-876f-b5023399dc69-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.186980    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81954e6-e83d-44b4-876f-b5023399dc69-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187013    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187044    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187068    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187094    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187115    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187137    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7818f918-9435-469c-a3b6-c16e509f15ea-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187161    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187186    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187235    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187276    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187379    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187406    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187447    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187471    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzkn\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-kube-api-access-zdzkn\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.208627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.187523    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrbkn\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-kube-api-access-vrbkn\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.294596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294270    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.294596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294317    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.294596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294358    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.294596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294381    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzkn\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-kube-api-access-zdzkn\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.294596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294453    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrbkn\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-kube-api-access-vrbkn\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.294596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294553    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7818f918-9435-469c-a3b6-c16e509f15ea-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294618    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81954e6-e83d-44b4-876f-b5023399dc69-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294689    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81954e6-e83d-44b4-876f-b5023399dc69-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294723    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294750    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294773    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294797    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294825    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294850    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7818f918-9435-469c-a3b6-c16e509f15ea-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294876    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294900    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294954    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.295040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.294999    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.295834    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.296128    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.325364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.326160    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7818f918-9435-469c-a3b6-c16e509f15ea-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.326409    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.330602    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81954e6-e83d-44b4-876f-b5023399dc69-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.338385    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.338903    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.339126    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.339402    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.341632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.339733    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.353821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.347475    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7818f918-9435-469c-a3b6-c16e509f15ea-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.353821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.351092    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.353821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.351322    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.353821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.351904    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81954e6-e83d-44b4-876f-b5023399dc69-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.353821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.352386    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.356793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.355511    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrbkn\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-kube-api-access-vrbkn\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.357506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.357476    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzkn\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-kube-api-access-zdzkn\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.357835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.357813    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.481784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.481752    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:30.520535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.520506    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:30.558435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.556096    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 10:44:30.558435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.556147    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 10:44:30.821991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.821958    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"]
Nov 03 10:44:30.823070 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:30.823043    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81954e6_e83d_44b4_876f_b5023399dc69.slice/crio-4845f858d5eb4ee9629db4d51264871461b62dbd3901f33c30b95c3b0c5201f5 WatchSource:0}: Error finding container 4845f858d5eb4ee9629db4d51264871461b62dbd3901f33c30b95c3b0c5201f5: Status 404 returned error can't find the container with id 4845f858d5eb4ee9629db4d51264871461b62dbd3901f33c30b95c3b0c5201f5
Nov 03 10:44:30.825598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.825559    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"]
Nov 03 10:44:30.827174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.827143    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:30.827271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.827201    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:30.827271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.827229    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:30.828020 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:30.827675    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7818f918_9435_469c_a3b6_c16e509f15ea.slice/crio-2564e5e98e78a078c1f797144219ff44b99901afe85e242d78a9e91169e93ffc WatchSource:0}: Error finding container 2564e5e98e78a078c1f797144219ff44b99901afe85e242d78a9e91169e93ffc: Status 404 returned error can't find the container with id 2564e5e98e78a078c1f797144219ff44b99901afe85e242d78a9e91169e93ffc
Nov 03 10:44:30.830426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.830402    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:30.830491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.830460    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:30.830527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:30.830499    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:31.086667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.086445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:31.238103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.238049    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" event={"ID":"7818f918-9435-469c-a3b6-c16e509f15ea","Type":"ContainerStarted","Data":"6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb"}
Nov 03 10:44:31.238504 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.238136    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" event={"ID":"7818f918-9435-469c-a3b6-c16e509f15ea","Type":"ContainerStarted","Data":"2564e5e98e78a078c1f797144219ff44b99901afe85e242d78a9e91169e93ffc"}
Nov 03 10:44:31.241009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.240982    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" event={"ID":"d81954e6-e83d-44b4-876f-b5023399dc69","Type":"ContainerStarted","Data":"792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459"}
Nov 03 10:44:31.241120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.241034    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" event={"ID":"d81954e6-e83d-44b4-876f-b5023399dc69","Type":"ContainerStarted","Data":"4845f858d5eb4ee9629db4d51264871461b62dbd3901f33c30b95c3b0c5201f5"}
Nov 03 10:44:31.262381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.261813    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podStartSLOduration=1.2617976610000001 podStartE2EDuration="1.261797661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:31.258357382 +0000 UTC m=+901.463939433" watchObservedRunningTime="2026-11-03 10:44:31.261797661 +0000 UTC m=+901.467379712"
Nov 03 10:44:31.283448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.283393    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podStartSLOduration=1.283375351 podStartE2EDuration="1.283375351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:31.279732986 +0000 UTC m=+901.485315037" watchObservedRunningTime="2026-11-03 10:44:31.283375351 +0000 UTC m=+901.488957404"
Nov 03 10:44:31.483089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.483008    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:31.485400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.485075    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 10:44:31.485400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.485134    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 10:44:31.524678 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.524635    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:31.529597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.526504    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:44:31.529597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.526560    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 10:44:31.558594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.556291    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 10:44:31.558594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:31.556349    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 10:44:32.246599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:32.243595    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:32.541655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:32.540046    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 10:44:32.541655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:32.540105    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 10:44:32.545594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:32.542392    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:44:32.545594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:32.542436    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 10:44:32.654598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:32.560902    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 10:44:32.654598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:32.560952    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 10:44:33.123593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.121210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:33.483073 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.483030    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 10:44:33.483469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.483095    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 10:44:33.525607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.521923    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:44:33.525607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.521984    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 10:44:33.559609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.558157    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 10:44:33.559609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.558224    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 10:44:33.770654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.769642    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628" podUID="b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" containerName="payload-processing" containerID="cri-o://9ad480e29171c66e4ce89d13a59783f93b410755f9453de7e913d6eba09c4c42" gracePeriod=25
Nov 03 10:44:33.777901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:33.777857    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c" podUID="6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" containerName="payload-processing" containerID="cri-o://d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa" gracePeriod=25
Nov 03 10:44:34.129681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.112673    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:34.485743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.485692    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 10:44:34.486381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.485755    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 10:44:34.544359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.544324    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:44:34.544688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.544644    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 10:44:34.568605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.564754    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:44:34.569886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.569840    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 10:44:34.570067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.570046    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 10:44:34.676654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.676624    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-routing-overlay-volume\") pod \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") "
Nov 03 10:44:34.676907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.676883    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-kube-api-access-p2854\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-kube-api-access-p2854\") pod \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") "
Nov 03 10:44:34.677091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.677075    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-provider-credentials\") pod \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") "
Nov 03 10:44:34.677610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.677587    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-plugins-config-volume\") pod \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\" (UID: \"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af\") "
Nov 03 10:44:34.678565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.678542    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-plugins-config-volume" pod "6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" (UID: "6a6db1dd-d8b2-4a95-bd52-4b44a7b611af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:34.679627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.679559    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-routing-overlay-volume" pod "6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" (UID: "6a6db1dd-d8b2-4a95-bd52-4b44a7b611af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:34.683115 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.683079    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-kube-api-access-p2854" pod "6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" (UID: "6a6db1dd-d8b2-4a95-bd52-4b44a7b611af"). InnerVolumeSpecName "kube-api-access-p2854". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:34.763527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.762935    5817 generic.go:356] "Generic (PLEG): container finished" podID="b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" containerID="9ad480e29171c66e4ce89d13a59783f93b410755f9453de7e913d6eba09c4c42" exitCode=0
Nov 03 10:44:34.763527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.763116    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628" event={"ID":"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022","Type":"ContainerDied","Data":"9ad480e29171c66e4ce89d13a59783f93b410755f9453de7e913d6eba09c4c42"}
Nov 03 10:44:34.766923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.766007    5817 generic.go:356] "Generic (PLEG): container finished" podID="6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" containerID="d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa" exitCode=0
Nov 03 10:44:34.766923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.766230    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c" event={"ID":"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af","Type":"ContainerDied","Data":"d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa"}
Nov 03 10:44:34.766923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.766277    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c" event={"ID":"6a6db1dd-d8b2-4a95-bd52-4b44a7b611af","Type":"ContainerDied","Data":"ee79099100995f964bb68f998156e96bdafc42c5bb7d9adb6274fe062621b632"}
Nov 03 10:44:34.766923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.766310    5817 scope.go:122] "RemoveContainer" containerID="d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa"
Nov 03 10:44:34.766923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.766475    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5944845765-fcr4c"
Nov 03 10:44:34.769118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.769000    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:34.772745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.772709    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-provider-credentials" pod "6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" (UID: "6a6db1dd-d8b2-4a95-bd52-4b44a7b611af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:34.779498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.778536    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-provider-credentials\") pod \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") "
Nov 03 10:44:34.779498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.778632    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-kube-api-access-6hgqq\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-kube-api-access-6hgqq\") pod \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") "
Nov 03 10:44:34.779498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.778697    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-routing-overlay-volume\") pod \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") "
Nov 03 10:44:34.779498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.778873    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-plugins-config-volume\") pod \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\" (UID: \"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022\") "
Nov 03 10:44:34.779498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.779252    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:34.779498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.779268    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:34.779498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.779284    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:34.779498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.779300    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2854\" (UniqueName: \"kubernetes.io/projected/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af-kube-api-access-p2854\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:34.779973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.779659    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-plugins-config-volume" pod "b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" (UID: "b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:34.784901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.784870    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-routing-overlay-volume" pod "b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" (UID: "b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:34.786227 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.786205    5817 scope.go:122] "RemoveContainer" containerID="d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa"
Nov 03 10:44:34.848609 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:44:34.837992    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa\": container with ID starting with d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa not found: ID does not exist" containerID="d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa"
Nov 03 10:44:34.848609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.838037    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa"} err="failed to get container status \"d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa\": rpc error: code = NotFound desc = could not find container \"d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa\": container with ID starting with d7e7d84fe805eaa91f84b13dfc2fdf8b96ba45cee6ee4b7346c62249524d57aa not found: ID does not exist"
Nov 03 10:44:34.866696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.866643    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-kube-api-access-6hgqq" pod "b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" (UID: "b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022"). InnerVolumeSpecName "kube-api-access-6hgqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:34.908709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.907631    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:34.908709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.907669    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgqq\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-kube-api-access-6hgqq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:34.908709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.907684    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:34.966771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:34.966711    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-provider-credentials" pod "b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" (UID: "b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:35.060245 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.060203    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:35.125720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.125639    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5944845765-fcr4c"]
Nov 03 10:44:35.140606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.139650    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5944845765-fcr4c"]
Nov 03 10:44:35.277739 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.277638    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:35.482322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.482294    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 10:44:35.482531 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.482510    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 10:44:35.525599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.525034    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:44:35.525599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.525111    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 10:44:35.784640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.784610    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628" event={"ID":"b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022","Type":"ContainerDied","Data":"009382091cb197009812b50baef57827c9a6e89021460c43717af056197181fc"}
Nov 03 10:44:35.784944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.784886    5817 scope.go:122] "RemoveContainer" containerID="9ad480e29171c66e4ce89d13a59783f93b410755f9453de7e913d6eba09c4c42"
Nov 03 10:44:35.785149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.785135    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74b7555969-r8628"
Nov 03 10:44:35.965598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.964794    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" path="/var/lib/kubelet/pods/6a6db1dd-d8b2-4a95-bd52-4b44a7b611af/volumes"
Nov 03 10:44:35.965598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.965231    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-r8628"]
Nov 03 10:44:35.965598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:35.965270    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-74b7555969-r8628"]
Nov 03 10:44:36.102656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:36.100146    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:36.102656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:36.101548    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:36.117638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:36.111827    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:44:36.420647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:36.420035    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:36.485597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:36.484006    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 10:44:36.485597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:36.484076    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 10:44:36.524372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:36.520926    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:44:36.524372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:36.520984    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 10:44:37.018647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.018615    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:37.160290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.160257    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:37.297794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.297703    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:37.482323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.482286    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 10:44:37.482465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.482352    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 10:44:37.521936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.521903    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:44:37.522209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.522184    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 10:44:37.663813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.663725    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:37.908510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:37.908475    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" path="/var/lib/kubelet/pods/b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022/volumes"
Nov 03 10:44:38.667286 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:38.667240    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:38.667861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:38.667842    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:38.668848 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:38.668829    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:44:38.700944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:38.700915    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:38.761725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:38.759758    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:38.763934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:38.762851    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:38.771371 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:38.766464    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:44:39.577697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:39.577253    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:40.453386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.453355    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:40.808743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.808662    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"]
Nov 03 10:44:40.809562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.809541    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" containerName="payload-processing"
Nov 03 10:44:40.809711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.809584    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" containerName="payload-processing"
Nov 03 10:44:40.809711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.809625    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" containerName="payload-processing"
Nov 03 10:44:40.809711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.809633    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" containerName="payload-processing"
Nov 03 10:44:40.809909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.809802    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6db1dd-d8b2-4a95-bd52-4b44a7b611af" containerName="payload-processing"
Nov 03 10:44:40.809909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.809822    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14bd8eb-44df-4d3b-b2a7-b4b8de9fc022" containerName="payload-processing"
Nov 03 10:44:40.825532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.823843    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.825532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.825496    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"]
Nov 03 10:44:40.828917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.828874    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"kube-root-ca.crt\""
Nov 03 10:44:40.828917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.828872    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"distinct-w0-f78621-kserve-self-signed-certs\""
Nov 03 10:44:40.828917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.828912    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"openshift-service-ca.crt\""
Nov 03 10:44:40.829133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.828925    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"default-dockercfg-rp5ss\""
Nov 03 10:44:40.894700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.894672    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-dshm\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.894874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.894706    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tls-certs\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.894874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.894853    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-home\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.894999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.894913    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2mls\" (UniqueName: \"kubernetes.io/projected/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-kube-api-access-v2mls\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.895050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.895009    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-model-cache\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.895050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.895042    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tmp-dir\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.904185 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.904158    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:40.996638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.996604    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-model-cache\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.996799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.996672    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tmp-dir\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.996799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.996722    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-dshm\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.996799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.996750    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tls-certs\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.996968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.996864    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-home\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.996968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.996921    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mls\" (UniqueName: \"kubernetes.io/projected/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-kube-api-access-v2mls\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.997077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.996977    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-model-cache\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.997077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.997046    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tmp-dir\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.997300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.997280    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-home\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.999308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.999281    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-dshm\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:40.999974 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:40.999953    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tls-certs\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:41.005427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.005405    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2mls\" (UniqueName: \"kubernetes.io/projected/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-kube-api-access-v2mls\") pod \"distinct-w0-f78621-kserve-78f698677c-fnr5w\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:41.114649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.113861    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"]
Nov 03 10:44:41.134592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.128489    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"]
Nov 03 10:44:41.134592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.129216    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.221596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.219605    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:44:41.221596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.220398    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:44:41.221596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.221366    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.221596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.221441    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.221596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.221530    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghvr\" (UniqueName: \"kubernetes.io/projected/7c3f1b62-c382-493f-a22a-22ad7f3a5584-kube-api-access-nghvr\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.227266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.227239    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:41.346813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.346771    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.347226 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.347181    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.347447 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.347432    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghvr\" (UniqueName: \"kubernetes.io/projected/7c3f1b62-c382-493f-a22a-22ad7f3a5584-kube-api-access-nghvr\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.448668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.427126    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.526709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.513463    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghvr\" (UniqueName: \"kubernetes.io/projected/7c3f1b62-c382-493f-a22a-22ad7f3a5584-kube-api-access-nghvr\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.608598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.604667    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b799193b-nx97m\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.708751 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.708117    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:41.712595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:41.709450    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:42.461599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.435658    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.498652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.461910    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:44:42.766251 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.766146    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.799724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.799668    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:44:42.846207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.846175    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"]
Nov 03 10:44:42.862719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.862010    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.863358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.863334    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.868733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.868708    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.874781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.874748    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.876209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.876173    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.882486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.882462    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.882677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.882659    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.882855 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.882840    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.882959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.882943    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f49022-14e6-474a-850e-def0cbe2e5a2-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.883128 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.883112    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.883283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.883269    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn6gb\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-kube-api-access-pn6gb\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:42.883383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:42.883370    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.046646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.027662    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.062675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.060235    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"]
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.066810    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.066910    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn6gb\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-kube-api-access-pn6gb\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.066944    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.067066    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.067166    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.067207    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.067235    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.067335    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.067359    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f49022-14e6-474a-850e-def0cbe2e5a2-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.068333    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f49022-14e6-474a-850e-def0cbe2e5a2-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.075709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.069291    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.188597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.187709    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.202596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.200221    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.209595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.207735    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.245183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.237828    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"]
Nov 03 10:44:43.339725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.317013    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:44:43.342190 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.340244    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.405452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.405399    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.414629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.407806    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.440705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.440657    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.488185 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.442305    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.546688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.498939    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.546688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.513643    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" event={"ID":"9feb7ae5-1475-4ffc-a636-2f8dd611ec28","Type":"ContainerStarted","Data":"b2a1336bab64a82be10b78f9a0ba7898d950f2a24a44f7e37ccd7facc7f121e3"}
Nov 03 10:44:43.546688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.538736    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:44:43.546688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.538790    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:44:43.887722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.612627    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn6gb\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-kube-api-access-pn6gb\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.887722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.747001    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"]
Nov 03 10:44:43.887722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.770669    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:44:43.887722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.770999    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:44:43.887722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.810720    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:43.887722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.839642    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.887722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.840311    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:44:43.947655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.892855    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.947655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:43.894775    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:44.065967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.065921    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.314615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.302483    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:44:44.314615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.302551    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.341604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.337506    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.356718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.356671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:44.437621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.416017    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.577652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.528680    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.636594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.529778    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.946215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.908146    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:44:44.946215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.908529    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:44:44.968293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:44.954803    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.128651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.100820    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:44:45.175766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.175723    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:44:45.355846 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:44:45.355806    5817 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.461s"
Nov 03 10:44:45.359748 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.359712    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m" event={"ID":"7c3f1b62-c382-493f-a22a-22ad7f3a5584","Type":"ContainerStarted","Data":"8f56f482011feb6a2800dffc6140ed1d1f25253e663f56c01e52d81ba47303f5"}
Nov 03 10:44:45.385008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.384928    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:45.385878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.385859    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:44:45.386032 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.386019    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:44:45.400597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.379065    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.586591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.417108    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.603605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.554538    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:44:45.648441 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.648348    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.766926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.761878    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x9t8q" podUID="550263cd-a336-4f27-abad-fafbbc1c61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.815599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.789444    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:45.949042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.916712    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:44:45.949656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:45.949614    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:44:46.150711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.150666    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:44:46.194155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.177671    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.209392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.209308    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:44:46.259984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.259950    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.260237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.258888    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.339911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.339863    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.344886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.344860    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.345133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.345093    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.522838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.522725    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" event={"ID":"22f49022-14e6-474a-850e-def0cbe2e5a2","Type":"ContainerStarted","Data":"c89260fe02be5051d8a422a4dc8b76f34d21e3dc305922486df5ee9b01d59018"}
Nov 03 10:44:46.595892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.595832    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:44:46.597507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.597457    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:44:46.601106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.600177    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.660599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.636480    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"]
Nov 03 10:44:46.698627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.687897    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.710602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.710384    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:46.804653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.804241    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 03 10:44:46.941771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.941713    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:44:47.003599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:46.980675    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:47.159702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.157523    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:47.180646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.180559    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:47.649725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.649663    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:44:47.656533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.655150    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:44:47.682480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.661224    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:44:47.716612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.713815    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:44:47.738715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.722875    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:47.829660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.826732    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:48.005595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.989308    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:44:48.005595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:47.990972    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:48.109596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.099871    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" podStartSLOduration=6.099847016 podStartE2EDuration="6.099847016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:48.073166593 +0000 UTC m=+918.278748647" watchObservedRunningTime="2026-11-03 10:44:48.099847016 +0000 UTC m=+918.305429068"
Nov 03 10:44:48.166246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.166208    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:44:48.310563 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.197451    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 03 10:44:48.313944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.197517    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:48.314388 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.313153    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" event={"ID":"22f49022-14e6-474a-850e-def0cbe2e5a2","Type":"ContainerStarted","Data":"d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a"}
Nov 03 10:44:48.314661 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.314643    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:44:48.314846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.314828    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m" event={"ID":"7c3f1b62-c382-493f-a22a-22ad7f3a5584","Type":"ContainerStarted","Data":"a03b70032e6ce130125dbe503453e2acc6ac7178b5c1bb9db7d41549bd55568e"}
Nov 03 10:44:48.315160 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.315146    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:44:48.399600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.376362    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m" podStartSLOduration=7.376343846 podStartE2EDuration="7.376343846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:48.164127304 +0000 UTC m=+918.369709357" watchObservedRunningTime="2026-11-03 10:44:48.376343846 +0000 UTC m=+918.581925901"
Nov 03 10:44:48.402644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.386681    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:44:48.477219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.477192    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:44:48.503008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.502979    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:48.611655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.580864    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:44:48.611951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.611931    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:44:48.634181 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.634125    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:44:48.651049 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.651023    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:44:48.840756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.840730    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:48.841809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.841788    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:44:48.841879 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:48.841834    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 10:44:49.063656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:49.063621    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" event={"ID":"9feb7ae5-1475-4ffc-a636-2f8dd611ec28","Type":"ContainerStarted","Data":"cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e"}
Nov 03 10:44:49.065323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:49.065300    5817 generic.go:356] "Generic (PLEG): container finished" podID="7c3f1b62-c382-493f-a22a-22ad7f3a5584" containerID="a03b70032e6ce130125dbe503453e2acc6ac7178b5c1bb9db7d41549bd55568e" exitCode=0
Nov 03 10:44:49.065415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:49.065345    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m" event={"ID":"7c3f1b62-c382-493f-a22a-22ad7f3a5584","Type":"ContainerDied","Data":"a03b70032e6ce130125dbe503453e2acc6ac7178b5c1bb9db7d41549bd55568e"}
Nov 03 10:44:49.079977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:49.079934    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podStartSLOduration=9.07992224 podStartE2EDuration="9.07992224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:49.079147025 +0000 UTC m=+919.284729076" watchObservedRunningTime="2026-11-03 10:44:49.07992224 +0000 UTC m=+919.285504291"
Nov 03 10:44:49.231610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:49.228897    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:49.618597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:49.618343    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:49.840639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:49.840604    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:44:49.840792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:49.840667    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 10:44:50.326845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.326785    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:50.355210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.355179    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-xks-ca\") pod \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") "
Nov 03 10:44:50.355374 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.355321    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3f1b62-c382-493f-a22a-22ad7f3a5584-kube-api-access-nghvr\" (UniqueName: \"kubernetes.io/projected/7c3f1b62-c382-493f-a22a-22ad7f3a5584-kube-api-access-nghvr\") pod \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") "
Nov 03 10:44:50.355434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.355379    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-openshift-ca\") pod \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\" (UID: \"7c3f1b62-c382-493f-a22a-22ad7f3a5584\") "
Nov 03 10:44:50.355814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.355787    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-openshift-ca" pod "7c3f1b62-c382-493f-a22a-22ad7f3a5584" (UID: "7c3f1b62-c382-493f-a22a-22ad7f3a5584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:50.355921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.355899    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:50.357441 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.357393    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-xks-ca" pod "7c3f1b62-c382-493f-a22a-22ad7f3a5584" (UID: "7c3f1b62-c382-493f-a22a-22ad7f3a5584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:50.357791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.357775    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3f1b62-c382-493f-a22a-22ad7f3a5584-kube-api-access-nghvr" pod "7c3f1b62-c382-493f-a22a-22ad7f3a5584" (UID: "7c3f1b62-c382-493f-a22a-22ad7f3a5584"). InnerVolumeSpecName "kube-api-access-nghvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:50.436341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.436301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:50.457146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.457112    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3f1b62-c382-493f-a22a-22ad7f3a5584-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:50.457146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.457148    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghvr\" (UniqueName: \"kubernetes.io/projected/7c3f1b62-c382-493f-a22a-22ad7f3a5584-kube-api-access-nghvr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:50.840363 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.840327    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:44:50.840544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:50.840393    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 10:44:51.081872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.081830    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m" event={"ID":"7c3f1b62-c382-493f-a22a-22ad7f3a5584","Type":"ContainerDied","Data":"8f56f482011feb6a2800dffc6140ed1d1f25253e663f56c01e52d81ba47303f5"}
Nov 03 10:44:51.081872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.081864    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"
Nov 03 10:44:51.082087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.081889    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f56f482011feb6a2800dffc6140ed1d1f25253e663f56c01e52d81ba47303f5"
Nov 03 10:44:51.227974 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.227943    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:51.241244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.241220    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:51.319232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.319199    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:51.767489 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.767446    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"]
Nov 03 10:44:51.768246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.768225    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7c3f1b62-c382-493f-a22a-22ad7f3a5584" containerName="revoke-keys"
Nov 03 10:44:51.768246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.768247    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3f1b62-c382-493f-a22a-22ad7f3a5584" containerName="revoke-keys"
Nov 03 10:44:51.768449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.768434    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3f1b62-c382-493f-a22a-22ad7f3a5584" containerName="revoke-keys"
Nov 03 10:44:51.771971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.771951    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.775887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.775864    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:44:51.776045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.776016    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:44:51.781293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.781265    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"]
Nov 03 10:44:51.840753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.840717    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:44:51.840937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.840781    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 10:44:51.875321 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.875285    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.875493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.875411    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z66p\" (UniqueName: \"kubernetes.io/projected/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-kube-api-access-9z66p\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.875493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.875461    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.976536 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.976504    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z66p\" (UniqueName: \"kubernetes.io/projected/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-kube-api-access-9z66p\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.976724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.976557    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.976799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.976775    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.977388 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.977363    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.979474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.979451    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:51.984301 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:51.984279    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z66p\" (UniqueName: \"kubernetes.io/projected/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-kube-api-access-9z66p\") pod \"maas-api-revoke-sub-9e2f8cca-8bqjg\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:52.115859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:52.115765    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:52.125663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:52.125636    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:44:52.367689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:52.357166    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:52.454020 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:52.453109    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"]
Nov 03 10:44:52.468744 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:44:52.468718    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:44:52.840421 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:52.840391    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:44:52.840923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:52.840446    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 10:44:53.122752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:53.122634    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg" event={"ID":"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3","Type":"ContainerStarted","Data":"31001e222ca2947932c0b048a18110f5a155a3a22eb56f63f44d840eebcfeea6"}
Nov 03 10:44:53.122752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:53.122721    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg" event={"ID":"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3","Type":"ContainerStarted","Data":"8c8d846d4fc1102a097859e334fb5c338a5a854697bdf5f8f064006c0805bd6a"}
Nov 03 10:44:53.144627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:53.144557    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg" podStartSLOduration=2.144537857 podStartE2EDuration="2.144537857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:53.138120487 +0000 UTC m=+923.343702535" watchObservedRunningTime="2026-11-03 10:44:53.144537857 +0000 UTC m=+923.350119909"
Nov 03 10:44:53.258949 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:53.258914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:53.840799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:53.840725    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:44:53.840799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:53.840777    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 10:44:54.074019 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:54.072549    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:54.134295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:54.134191    5817 generic.go:356] "Generic (PLEG): container finished" podID="b08feb4f-b2e1-4f4e-b23d-3859a661bdd3" containerID="31001e222ca2947932c0b048a18110f5a155a3a22eb56f63f44d840eebcfeea6" exitCode=0
Nov 03 10:44:54.134474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:54.134296    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg" event={"ID":"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3","Type":"ContainerDied","Data":"31001e222ca2947932c0b048a18110f5a155a3a22eb56f63f44d840eebcfeea6"}
Nov 03 10:44:54.845945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:54.845913    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:54.846392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:54.846368    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:54.847287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:54.847265    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:44:54.958752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:54.958723    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:55.308105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.308075    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:55.446852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.446825    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-xks-ca\") pod \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") "
Nov 03 10:44:55.447012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.446885    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-kube-api-access-9z66p\" (UniqueName: \"kubernetes.io/projected/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-kube-api-access-9z66p\") pod \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") "
Nov 03 10:44:55.447012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.446945    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-openshift-ca\") pod \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\" (UID: \"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3\") "
Nov 03 10:44:55.447424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.447386    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-openshift-ca" pod "b08feb4f-b2e1-4f4e-b23d-3859a661bdd3" (UID: "b08feb4f-b2e1-4f4e-b23d-3859a661bdd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:55.447562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.447520    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:55.449155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.449124    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-xks-ca" pod "b08feb4f-b2e1-4f4e-b23d-3859a661bdd3" (UID: "b08feb4f-b2e1-4f4e-b23d-3859a661bdd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:55.449374 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.449354    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-kube-api-access-9z66p" pod "b08feb4f-b2e1-4f4e-b23d-3859a661bdd3" (UID: "b08feb4f-b2e1-4f4e-b23d-3859a661bdd3"). InnerVolumeSpecName "kube-api-access-9z66p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:55.548477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.548451    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:55.548477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.548475    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z66p\" (UniqueName: \"kubernetes.io/projected/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3-kube-api-access-9z66p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:55.756352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:55.756201    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:56.153838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:56.153759    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"
Nov 03 10:44:56.153838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:56.153788    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg" event={"ID":"b08feb4f-b2e1-4f4e-b23d-3859a661bdd3","Type":"ContainerDied","Data":"8c8d846d4fc1102a097859e334fb5c338a5a854697bdf5f8f064006c0805bd6a"}
Nov 03 10:44:56.154280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:56.153839    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c8d846d4fc1102a097859e334fb5c338a5a854697bdf5f8f064006c0805bd6a"
Nov 03 10:44:56.582092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:56.582059    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:57.222550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.222509    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"]
Nov 03 10:44:57.223631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.223607    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b08feb4f-b2e1-4f4e-b23d-3859a661bdd3" containerName="revoke-keys"
Nov 03 10:44:57.223631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.223631    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08feb4f-b2e1-4f4e-b23d-3859a661bdd3" containerName="revoke-keys"
Nov 03 10:44:57.223862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.223849    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08feb4f-b2e1-4f4e-b23d-3859a661bdd3" containerName="revoke-keys"
Nov 03 10:44:57.227878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.227858    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.236812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.236792    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"distinct-2-w0-f78621-kserve-self-signed-certs\""
Nov 03 10:44:57.244566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.244544    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"]
Nov 03 10:44:57.308644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.308610    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:57.374802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.374711    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njjv7\" (UniqueName: \"kubernetes.io/projected/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-kube-api-access-njjv7\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.374802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.374778    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-dshm\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.375083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.374859    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-home\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.375083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.374877    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tmp-dir\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.375083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.374900    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tls-certs\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.375083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.374919    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-model-cache\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.442528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.442490    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:57.476689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.476602    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njjv7\" (UniqueName: \"kubernetes.io/projected/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-kube-api-access-njjv7\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.476689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.476660    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-dshm\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.476891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.476729    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-home\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.476891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.476755    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tmp-dir\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.476891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.476777    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tls-certs\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.476891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.476803    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-model-cache\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.477168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.477148    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tmp-dir\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.477234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.477196    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-home\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.477297 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.477274    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-model-cache\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.479381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.479359    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-dshm\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.479720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.479697    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tls-certs\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.485374 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.485353    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njjv7\" (UniqueName: \"kubernetes.io/projected/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-kube-api-access-njjv7\") pod \"distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") " pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.553182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.553149    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:57.663692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.662730    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"]
Nov 03 10:44:57.669659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.669635    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.683970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.683939    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"]
Nov 03 10:44:57.781096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781009    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.781096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781074    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.781315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781150    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.781315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781195    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdt7\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-kube-api-access-mgdt7\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.781315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781231    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.781315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781248    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.781315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781274    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.781598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781319    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.781598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.781407    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.846659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.846629    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"]
Nov 03 10:44:57.882803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.882775    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.882941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.882819    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.882941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.882852    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.882941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.882893    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.882941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.882921    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdt7\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-kube-api-access-mgdt7\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.882962    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.882982    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.883006    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.883056    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.883207    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.883266    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.883411    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.883551    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.883657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.883612    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.886100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.886078    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.886324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.886305    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.892381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.892352    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdt7\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-kube-api-access-mgdt7\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.893235 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.893201    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:57.996226 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:57.996192    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:44:58.173744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.173704    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" event={"ID":"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09","Type":"ContainerStarted","Data":"96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094"}
Nov 03 10:44:58.173895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.173757    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" event={"ID":"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09","Type":"ContainerStarted","Data":"c1e8769c663f95058823c09895e080f57096c75d7ccc59f8df84a80c5f2370e7"}
Nov 03 10:44:58.251567 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.251531    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:58.281012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.280599    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podStartSLOduration=1.280563543 podStartE2EDuration="1.280563543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:58.190965221 +0000 UTC m=+928.396547276" watchObservedRunningTime="2026-11-03 10:44:58.280563543 +0000 UTC m=+928.486145596"
Nov 03 10:44:58.284789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.284770    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"]
Nov 03 10:44:58.285223 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:44:58.285197    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06ebb7c_9841_4fb0_a7d5_8f945814e368.slice/crio-d3bc2300f5e6101c75a8a41d63cd60d3e26b72bcec49d87aee6d22ba19bd79b1 WatchSource:0}: Error finding container d3bc2300f5e6101c75a8a41d63cd60d3e26b72bcec49d87aee6d22ba19bd79b1: Status 404 returned error can't find the container with id d3bc2300f5e6101c75a8a41d63cd60d3e26b72bcec49d87aee6d22ba19bd79b1
Nov 03 10:44:58.287929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.287893    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:58.288043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.287972    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:58.288043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.288015    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:58.553319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:58.553281    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:44:59.108552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:59.108260    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:44:59.153663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:59.144081    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:44:59.184634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:59.184594    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" event={"ID":"e06ebb7c-9841-4fb0-a7d5-8f945814e368","Type":"ContainerStarted","Data":"07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4"}
Nov 03 10:44:59.184818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:59.184656    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" event={"ID":"e06ebb7c-9841-4fb0-a7d5-8f945814e368","Type":"ContainerStarted","Data":"d3bc2300f5e6101c75a8a41d63cd60d3e26b72bcec49d87aee6d22ba19bd79b1"}
Nov 03 10:44:59.227978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:44:59.227914    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podStartSLOduration=2.227895708 podStartE2EDuration="2.227895708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:59.225204586 +0000 UTC m=+929.430786658" watchObservedRunningTime="2026-11-03 10:44:59.227895708 +0000 UTC m=+929.433477762"
Nov 03 10:45:00.028600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.022660    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:45:00.028600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.022867    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:00.028600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.022912    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:00.232290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.229935    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"]
Nov 03 10:45:00.283159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.283075    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:00.310179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.310145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:00.329842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.329814    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:45:00.439605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.399439    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"]
Nov 03 10:45:00.720620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.716911    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19932057-de3d-45b1-b4ea-a64c2019943d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z\" (UID: \"19932057-de3d-45b1-b4ea-a64c2019943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:00.720620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.717121    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kvr\" (UniqueName: \"kubernetes.io/projected/19932057-de3d-45b1-b4ea-a64c2019943d-kube-api-access-q2kvr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z\" (UID: \"19932057-de3d-45b1-b4ea-a64c2019943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:00.740596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.739040    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"]
Nov 03 10:45:00.798591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.762744    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:00.835600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.808229    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"]
Nov 03 10:45:00.835600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.808291    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"]
Nov 03 10:45:00.835600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.808413    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:00.835600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.819637    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kvr\" (UniqueName: \"kubernetes.io/projected/19932057-de3d-45b1-b4ea-a64c2019943d-kube-api-access-q2kvr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z\" (UID: \"19932057-de3d-45b1-b4ea-a64c2019943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:00.835600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.819995    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19932057-de3d-45b1-b4ea-a64c2019943d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z\" (UID: \"19932057-de3d-45b1-b4ea-a64c2019943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:00.870601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.860471    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"]
Nov 03 10:45:00.896617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.891460    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19932057-de3d-45b1-b4ea-a64c2019943d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z\" (UID: \"19932057-de3d-45b1-b4ea-a64c2019943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:00.962602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.955112    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a820-25cb-4122-bc8b-3968b7252e34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l\" (UID: \"6eb8a820-25cb-4122-bc8b-3968b7252e34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:00.962602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.955168    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv\" (UID: \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:00.962602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.955313    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6h5f\" (UniqueName: \"kubernetes.io/projected/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-kube-api-access-v6h5f\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv\" (UID: \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:00.962602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:00.955360    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2457\" (UniqueName: \"kubernetes.io/projected/6eb8a820-25cb-4122-bc8b-3968b7252e34-kube-api-access-b2457\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l\" (UID: \"6eb8a820-25cb-4122-bc8b-3968b7252e34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:01.020860 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.015188    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.031598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.021907    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kvr\" (UniqueName: \"kubernetes.io/projected/19932057-de3d-45b1-b4ea-a64c2019943d-kube-api-access-q2kvr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z\" (UID: \"19932057-de3d-45b1-b4ea-a64c2019943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:01.061612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.040675    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:01.061612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.040731    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:01.104138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.104096    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.135598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.125475    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a820-25cb-4122-bc8b-3968b7252e34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l\" (UID: \"6eb8a820-25cb-4122-bc8b-3968b7252e34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:01.135598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.125530    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv\" (UID: \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:01.139595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.136660    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6h5f\" (UniqueName: \"kubernetes.io/projected/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-kube-api-access-v6h5f\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv\" (UID: \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:01.139595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.136732    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2457\" (UniqueName: \"kubernetes.io/projected/6eb8a820-25cb-4122-bc8b-3968b7252e34-kube-api-access-b2457\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l\" (UID: \"6eb8a820-25cb-4122-bc8b-3968b7252e34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:01.139595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.139302    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a820-25cb-4122-bc8b-3968b7252e34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l\" (UID: \"6eb8a820-25cb-4122-bc8b-3968b7252e34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:01.143597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.139917    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv\" (UID: \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:01.160689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.151163    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2457\" (UniqueName: \"kubernetes.io/projected/6eb8a820-25cb-4122-bc8b-3968b7252e34-kube-api-access-b2457\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l\" (UID: \"6eb8a820-25cb-4122-bc8b-3968b7252e34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:01.200937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.200541    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6h5f\" (UniqueName: \"kubernetes.io/projected/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-kube-api-access-v6h5f\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv\" (UID: \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:01.200937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.200675    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:45:01.200937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.200924    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:01.219620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.216491    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.393098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.392831    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:01.393098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.392926    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:01.592072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.591967    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.605896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.605840    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.710727 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.699672    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.729612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:01.716644    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.023118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.023085    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:02.026593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.023402    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:02.052597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.027756    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.081603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.071360    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:45:02.347050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.140548    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.347050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.141086    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:45:02.347050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.204103    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.347050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.204632    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:45:02.347050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.226615    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.347050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.227142    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:45:02.347050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.270490    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": context deadline exceeded"
Nov 03 10:45:02.684080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.638772    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.764736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.650523    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:45:02.764736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.740934    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:45:02.764736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.740997    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:45:02.793256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.785674    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.793256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.789045    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:45:03.067558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.794654    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:45:03.067558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.794703    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"]
Nov 03 10:45:03.067558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.794767    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:45:03.067558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:02.842651    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.067558 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:02.897639    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19932057_de3d_45b1_b4ea_a64c2019943d.slice/crio-ce2062d1f423438053f586ef50999746850a8f618b65252b9d27418d290dc44d WatchSource:0}: Error finding container ce2062d1f423438053f586ef50999746850a8f618b65252b9d27418d290dc44d: Status 404 returned error can't find the container with id ce2062d1f423438053f586ef50999746850a8f618b65252b9d27418d290dc44d
Nov 03 10:45:03.067558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.027830    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.138750 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.138417    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:03.195599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.186148    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:03.195599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.186203    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:03.195599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.187397    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.247660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.227904    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.247660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.231247    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:45:03.511603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.491344    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:03.821597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.780519    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.821597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.780949    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:03.932703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.900864    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:45:03.932703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:03.901314    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:45:04.039424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.039389    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:04.039718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.039685    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:04.040602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.039875    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.151702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.150681    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.327618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.322158    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:45:04.337602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.322551    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.337602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.328367    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:45:04.825791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.825744    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:04.827624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.827564    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:45:05.021993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:04.996420    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.094598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.051545    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.094598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.063545    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:45:05.094598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.067765    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.208159 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:05.208124    5817 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.271s"
Nov 03 10:45:05.208803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.208783    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:45:05.209163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.209138    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" event={"ID":"19932057-de3d-45b1-b4ea-a64c2019943d","Type":"ContainerStarted","Data":"ce2062d1f423438053f586ef50999746850a8f618b65252b9d27418d290dc44d"}
Nov 03 10:45:05.221645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.212992    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:05.239306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.235721    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:05.239306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.235859    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:05.374427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.357178    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.376706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.376660    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:45:05.626788 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.621796    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.826195 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.826149    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.828248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.828216    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.877161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.874378    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:05.884775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:05.882720    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:45:06.069063 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.066724    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:06.092800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.079818    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:06.092800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.079868    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:06.092800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.090916    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:06.179669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.179647    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:45:06.179904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.179892    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:45:06.180008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.179997    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"]
Nov 03 10:45:06.180146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.180136    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:45:06.180237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.180228    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"]
Nov 03 10:45:06.180351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.180342    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:45:06.180452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.180443    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:45:06.180544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.180535    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:45:06.180667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.180658    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:45:06.186440 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.186413    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" event={"ID":"6eb8a820-25cb-4122-bc8b-3968b7252e34","Type":"ContainerStarted","Data":"df4839c9d7c78cd29bb86a4fdb4757179ca7147eb804d318f9d7bca4eaa2cd28"}
Nov 03 10:45:06.240599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.237320    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:45:06.241038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.240739    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" event={"ID":"a2f6acfc-e13b-4eb9-ab55-71d8242cde12","Type":"ContainerStarted","Data":"f707f7fddeb0e71371765f9bb6841c8a4e476994104eae832f116fba77438693"}
Nov 03 10:45:06.376873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:06.376844    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:07.001011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.000922    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:07.001264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.001233    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:07.144844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.144805    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:07.264988 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.264898    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" event={"ID":"a2f6acfc-e13b-4eb9-ab55-71d8242cde12","Type":"ContainerStarted","Data":"9323294822cd5caf9c44d1a40aa201889e8d95f79a5bb10ac6ad1d24b286685d"}
Nov 03 10:45:07.266750 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.266718    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" event={"ID":"6eb8a820-25cb-4122-bc8b-3968b7252e34","Type":"ContainerStarted","Data":"619abcb5c3bf521da2507b4805dcbbc6f9cf9843ba74eeafcfb934f397074815"}
Nov 03 10:45:07.268386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.268361    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" event={"ID":"19932057-de3d-45b1-b4ea-a64c2019943d","Type":"ContainerStarted","Data":"42e938dc4ab53e1675912130db30ed133b1ffceff7b05101d173389746609b96"}
Nov 03 10:45:07.285184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.285144    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" podStartSLOduration=7.285133519 podStartE2EDuration="7.285133519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:07.281605699 +0000 UTC m=+937.487187750" watchObservedRunningTime="2026-11-03 10:45:07.285133519 +0000 UTC m=+937.490715566"
Nov 03 10:45:07.327932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.327883    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" podStartSLOduration=7.327864132 podStartE2EDuration="7.327864132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:07.302654229 +0000 UTC m=+937.508236281" watchObservedRunningTime="2026-11-03 10:45:07.327864132 +0000 UTC m=+937.533446185"
Nov 03 10:45:07.333366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.333329    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" podStartSLOduration=7.333315487 podStartE2EDuration="7.333315487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:07.321891903 +0000 UTC m=+937.527473959" watchObservedRunningTime="2026-11-03 10:45:07.333315487 +0000 UTC m=+937.538897537"
Nov 03 10:45:07.553954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.553876    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:45:07.566794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.566768    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:45:07.981549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.981512    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:08.006597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.999226    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:08.006597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:07.999280    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:08.278691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.278607    5817 generic.go:356] "Generic (PLEG): container finished" podID="19932057-de3d-45b1-b4ea-a64c2019943d" containerID="42e938dc4ab53e1675912130db30ed133b1ffceff7b05101d173389746609b96" exitCode=60
Nov 03 10:45:08.279197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.278699    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" event={"ID":"19932057-de3d-45b1-b4ea-a64c2019943d","Type":"ContainerDied","Data":"42e938dc4ab53e1675912130db30ed133b1ffceff7b05101d173389746609b96"}
Nov 03 10:45:08.279197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.279001    5817 scope.go:122] "RemoveContainer" containerID="42e938dc4ab53e1675912130db30ed133b1ffceff7b05101d173389746609b96"
Nov 03 10:45:08.280841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.280729    5817 generic.go:356] "Generic (PLEG): container finished" podID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerID="9323294822cd5caf9c44d1a40aa201889e8d95f79a5bb10ac6ad1d24b286685d" exitCode=60
Nov 03 10:45:08.280841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.280808    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" event={"ID":"a2f6acfc-e13b-4eb9-ab55-71d8242cde12","Type":"ContainerDied","Data":"9323294822cd5caf9c44d1a40aa201889e8d95f79a5bb10ac6ad1d24b286685d"}
Nov 03 10:45:08.281117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.281100    5817 scope.go:122] "RemoveContainer" containerID="9323294822cd5caf9c44d1a40aa201889e8d95f79a5bb10ac6ad1d24b286685d"
Nov 03 10:45:08.282867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.282664    5817 generic.go:356] "Generic (PLEG): container finished" podID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerID="619abcb5c3bf521da2507b4805dcbbc6f9cf9843ba74eeafcfb934f397074815" exitCode=60
Nov 03 10:45:08.282867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.282745    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" event={"ID":"6eb8a820-25cb-4122-bc8b-3968b7252e34","Type":"ContainerDied","Data":"619abcb5c3bf521da2507b4805dcbbc6f9cf9843ba74eeafcfb934f397074815"}
Nov 03 10:45:08.283080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.283010    5817 scope.go:122] "RemoveContainer" containerID="619abcb5c3bf521da2507b4805dcbbc6f9cf9843ba74eeafcfb934f397074815"
Nov 03 10:45:08.328787 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.328761    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:45:08.767468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.767432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:08.998378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.998346    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 10:45:08.998564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:08.998399    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 10:45:09.298812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:09.298778    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" event={"ID":"a2f6acfc-e13b-4eb9-ab55-71d8242cde12","Type":"ContainerStarted","Data":"3d24754f6cb2c19f9369380b23aad2af00c4ae754785b6f5b9c1b3fa6c11ba00"}
Nov 03 10:45:09.301488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:09.301460    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" event={"ID":"6eb8a820-25cb-4122-bc8b-3968b7252e34","Type":"ContainerStarted","Data":"2241a44b21deba3ca53b53f7fe7c7b718d328f4c5f9acda9fefbd7bf14702d52"}
Nov 03 10:45:09.303855 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:09.303832    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" event={"ID":"19932057-de3d-45b1-b4ea-a64c2019943d","Type":"ContainerStarted","Data":"8537008c6e2eaf24a6aa2faf3d1ddf47a6eeab1c38f787bfaa01a1af1cb6f353"}
Nov 03 10:45:09.559448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:09.559304    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:10.317509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.317475    5817 generic.go:356] "Generic (PLEG): container finished" podID="19932057-de3d-45b1-b4ea-a64c2019943d" containerID="8537008c6e2eaf24a6aa2faf3d1ddf47a6eeab1c38f787bfaa01a1af1cb6f353" exitCode=60
Nov 03 10:45:10.318016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.317706    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" event={"ID":"19932057-de3d-45b1-b4ea-a64c2019943d","Type":"ContainerDied","Data":"8537008c6e2eaf24a6aa2faf3d1ddf47a6eeab1c38f787bfaa01a1af1cb6f353"}
Nov 03 10:45:10.318016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.317872    5817 scope.go:122] "RemoveContainer" containerID="42e938dc4ab53e1675912130db30ed133b1ffceff7b05101d173389746609b96"
Nov 03 10:45:10.318016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.317960    5817 scope.go:122] "RemoveContainer" containerID="8537008c6e2eaf24a6aa2faf3d1ddf47a6eeab1c38f787bfaa01a1af1cb6f353"
Nov 03 10:45:10.318976 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:10.318945    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z_odh-ai-gateway-infra(19932057-de3d-45b1-b4ea-a64c2019943d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" podUID="19932057-de3d-45b1-b4ea-a64c2019943d"
Nov 03 10:45:10.321388 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.321366    5817 generic.go:356] "Generic (PLEG): container finished" podID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerID="3d24754f6cb2c19f9369380b23aad2af00c4ae754785b6f5b9c1b3fa6c11ba00" exitCode=60
Nov 03 10:45:10.321654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.321399    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" event={"ID":"a2f6acfc-e13b-4eb9-ab55-71d8242cde12","Type":"ContainerDied","Data":"3d24754f6cb2c19f9369380b23aad2af00c4ae754785b6f5b9c1b3fa6c11ba00"}
Nov 03 10:45:10.321763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.321682    5817 scope.go:122] "RemoveContainer" containerID="3d24754f6cb2c19f9369380b23aad2af00c4ae754785b6f5b9c1b3fa6c11ba00"
Nov 03 10:45:10.322025 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:10.321934    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv_odh-ai-gateway-infra(a2f6acfc-e13b-4eb9-ab55-71d8242cde12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12"
Nov 03 10:45:10.324334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.324312    5817 generic.go:356] "Generic (PLEG): container finished" podID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerID="2241a44b21deba3ca53b53f7fe7c7b718d328f4c5f9acda9fefbd7bf14702d52" exitCode=60
Nov 03 10:45:10.324431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.324368    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" event={"ID":"6eb8a820-25cb-4122-bc8b-3968b7252e34","Type":"ContainerDied","Data":"2241a44b21deba3ca53b53f7fe7c7b718d328f4c5f9acda9fefbd7bf14702d52"}
Nov 03 10:45:10.324773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.324748    5817 scope.go:122] "RemoveContainer" containerID="2241a44b21deba3ca53b53f7fe7c7b718d328f4c5f9acda9fefbd7bf14702d52"
Nov 03 10:45:10.325015 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:10.324992    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l_odh-ai-gateway-infra(6eb8a820-25cb-4122-bc8b-3968b7252e34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34"
Nov 03 10:45:10.339173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.339108    5817 scope.go:122] "RemoveContainer" containerID="9323294822cd5caf9c44d1a40aa201889e8d95f79a5bb10ac6ad1d24b286685d"
Nov 03 10:45:10.378800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.367747    5817 scope.go:122] "RemoveContainer" containerID="619abcb5c3bf521da2507b4805dcbbc6f9cf9843ba74eeafcfb934f397074815"
Nov 03 10:45:10.391594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.385772    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:10.849765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.849723    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:45:10.850214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.850194    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:45:10.851076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:10.851054    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 10:45:11.346590 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:11.343808    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:11.346590 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:11.345036    5817 scope.go:122] "RemoveContainer" containerID="3d24754f6cb2c19f9369380b23aad2af00c4ae754785b6f5b9c1b3fa6c11ba00"
Nov 03 10:45:11.346590 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:11.345310    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv_odh-ai-gateway-infra(a2f6acfc-e13b-4eb9-ab55-71d8242cde12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12"
Nov 03 10:45:11.348773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:11.347626    5817 scope.go:122] "RemoveContainer" containerID="2241a44b21deba3ca53b53f7fe7c7b718d328f4c5f9acda9fefbd7bf14702d52"
Nov 03 10:45:11.348773 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:11.347941    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l_odh-ai-gateway-infra(6eb8a820-25cb-4122-bc8b-3968b7252e34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34"
Nov 03 10:45:11.350889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:11.350864    5817 scope.go:122] "RemoveContainer" containerID="8537008c6e2eaf24a6aa2faf3d1ddf47a6eeab1c38f787bfaa01a1af1cb6f353"
Nov 03 10:45:11.351160 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:11.351136    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z_odh-ai-gateway-infra(19932057-de3d-45b1-b4ea-a64c2019943d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" podUID="19932057-de3d-45b1-b4ea-a64c2019943d"
Nov 03 10:45:12.071207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.071159    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:12.223615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.223565    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"]
Nov 03 10:45:12.230181 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.230155    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.235084 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.234649    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:12.237199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.237174    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"]
Nov 03 10:45:12.356453 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.356387    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.356906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.356451    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.356906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.356531    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xntc\" (UniqueName: \"kubernetes.io/projected/948d9d84-85d4-4a10-9873-674500d7ff77-kube-api-access-2xntc\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.410846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.410809    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"]
Nov 03 10:45:12.415353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.415331    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.442663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.442633    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"]
Nov 03 10:45:12.451461 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.451431    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"]
Nov 03 10:45:12.460815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.460791    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.463310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.463276    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.463430 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.463398    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.463500 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.463465    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xntc\" (UniqueName: \"kubernetes.io/projected/948d9d84-85d4-4a10-9873-674500d7ff77-kube-api-access-2xntc\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.464783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.464761    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.465214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.465187    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"]
Nov 03 10:45:12.468403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.468380    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.472985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.472958    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xntc\" (UniqueName: \"kubernetes.io/projected/948d9d84-85d4-4a10-9873-674500d7ff77-kube-api-access-2xntc\") pod \"maas-api-revoke-sub-44884449-pbpb6\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.550551 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.550526    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:12.565448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565305    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlv8\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-kube-api-access-pmlv8\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.565448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565359    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.565588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565510    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.565653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565585    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.565699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565648    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.565699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565674    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.565798 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565713    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdqc\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-kube-api-access-dmdqc\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.566007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565878    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.566007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565943    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.566007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.565984    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.566221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.566027    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541c102-d0ff-41af-805a-a15bc6951759-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.566221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.566115    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.566221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.566214    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.566369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.566326    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541c102-d0ff-41af-805a-a15bc6951759-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.566425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.566370    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.566488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.566443    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.566488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.566481    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.566638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.566522    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.667461 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667426    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlv8\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-kube-api-access-pmlv8\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.667656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667483    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.667656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667535    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.667656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667582    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.667656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667644    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.667952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667673    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.667952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667708    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdqc\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-kube-api-access-dmdqc\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.667952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667735    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.667952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667799    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.667952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667822    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.667952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667846    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541c102-d0ff-41af-805a-a15bc6951759-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.667952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667890    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.668323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667951    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.668323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.667981    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.668323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668012    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541c102-d0ff-41af-805a-a15bc6951759-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.668323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668065    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.668323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668106    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.668323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668147    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.668323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668187    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.668323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668278    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.668785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668516    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.668785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668544    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.668892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668825    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.668996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.668968    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.669207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.669184    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541c102-d0ff-41af-805a-a15bc6951759-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.669487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.669464    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.669738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.669717    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.669885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.669869    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.672711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.671777    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.672711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.671926    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.672711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.672198    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541c102-d0ff-41af-805a-a15bc6951759-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.677617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.673957    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.677617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.675417    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlv8\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-kube-api-access-pmlv8\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.677617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.677114    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdqc\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-kube-api-access-dmdqc\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.677617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.677160    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.677617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.677215    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.746512 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.746482    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:12.808622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.803455    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:12.844436 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.844408    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"]
Nov 03 10:45:12.845721 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:12.845691    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948d9d84_85d4_4a10_9873_674500d7ff77.slice/crio-7412898220d42bc7d5c65dc7237e8935a192559ab3b684c3db2da514aff04751 WatchSource:0}: Error finding container 7412898220d42bc7d5c65dc7237e8935a192559ab3b684c3db2da514aff04751: Status 404 returned error can't find the container with id 7412898220d42bc7d5c65dc7237e8935a192559ab3b684c3db2da514aff04751
Nov 03 10:45:12.973656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:12.973544    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:13.044365 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.043466    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"]
Nov 03 10:45:13.046305 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:13.046278    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1541c102_d0ff_41af_805a_a15bc6951759.slice/crio-d80d6b6a41162005c16cf912539bd4c3dbbbdde89adfd76333336d435b1800f7 WatchSource:0}: Error finding container d80d6b6a41162005c16cf912539bd4c3dbbbdde89adfd76333336d435b1800f7: Status 404 returned error can't find the container with id d80d6b6a41162005c16cf912539bd4c3dbbbdde89adfd76333336d435b1800f7
Nov 03 10:45:13.049125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.049094    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.049199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.049157    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.049199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.049185    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.129357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.129329    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"]
Nov 03 10:45:13.129990 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:13.129967    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdda72df_d1e1_4ab8_9665_8bbcb2f2f9ec.slice/crio-74f0131c2af8cb306d57468f0d5d967eb2c3807380fd2562a395f447d253e613 WatchSource:0}: Error finding container 74f0131c2af8cb306d57468f0d5d967eb2c3807380fd2562a395f447d253e613: Status 404 returned error can't find the container with id 74f0131c2af8cb306d57468f0d5d967eb2c3807380fd2562a395f447d253e613
Nov 03 10:45:13.133267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.133225    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.133373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.133311    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.133373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.133355    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.369506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.369413    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6" event={"ID":"948d9d84-85d4-4a10-9873-674500d7ff77","Type":"ContainerStarted","Data":"fb8da1ab2fe5e7d6210820d78433278cf0baf905777ab20d8d08e22a0005737d"}
Nov 03 10:45:13.369506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.369493    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6" event={"ID":"948d9d84-85d4-4a10-9873-674500d7ff77","Type":"ContainerStarted","Data":"7412898220d42bc7d5c65dc7237e8935a192559ab3b684c3db2da514aff04751"}
Nov 03 10:45:13.371638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.371615    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" event={"ID":"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec","Type":"ContainerStarted","Data":"560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4"}
Nov 03 10:45:13.371779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.371664    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" event={"ID":"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec","Type":"ContainerStarted","Data":"74f0131c2af8cb306d57468f0d5d967eb2c3807380fd2562a395f447d253e613"}
Nov 03 10:45:13.373632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.373605    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" event={"ID":"1541c102-d0ff-41af-805a-a15bc6951759","Type":"ContainerStarted","Data":"7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d"}
Nov 03 10:45:13.373757 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.373650    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" event={"ID":"1541c102-d0ff-41af-805a-a15bc6951759","Type":"ContainerStarted","Data":"d80d6b6a41162005c16cf912539bd4c3dbbbdde89adfd76333336d435b1800f7"}
Nov 03 10:45:13.404545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.404477    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6" podStartSLOduration=1.404458445 podStartE2EDuration="1.404458445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:13.383650874 +0000 UTC m=+943.589232925" watchObservedRunningTime="2026-11-03 10:45:13.404458445 +0000 UTC m=+943.610040497"
Nov 03 10:45:13.420723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.420631    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podStartSLOduration=1.42061378 podStartE2EDuration="1.42061378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:13.416258617 +0000 UTC m=+943.621840669" watchObservedRunningTime="2026-11-03 10:45:13.42061378 +0000 UTC m=+943.626195832"
Nov 03 10:45:13.438588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.438517    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podStartSLOduration=1.438500023 podStartE2EDuration="1.438500023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:13.435886459 +0000 UTC m=+943.641468510" watchObservedRunningTime="2026-11-03 10:45:13.438500023 +0000 UTC m=+943.644082075"
Nov 03 10:45:13.736668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.736634    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:13.749649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.746670    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:13.811407 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:13.811357    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:14.383585 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:14.383548    5817 generic.go:356] "Generic (PLEG): container finished" podID="948d9d84-85d4-4a10-9873-674500d7ff77" containerID="fb8da1ab2fe5e7d6210820d78433278cf0baf905777ab20d8d08e22a0005737d" exitCode=0
Nov 03 10:45:14.383963 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:14.383639    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6" event={"ID":"948d9d84-85d4-4a10-9873-674500d7ff77","Type":"ContainerDied","Data":"fb8da1ab2fe5e7d6210820d78433278cf0baf905777ab20d8d08e22a0005737d"}
Nov 03 10:45:14.458088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:14.454165    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:14.757881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:14.757830    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 10:45:14.758166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:14.758139    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 10:45:14.807606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:14.807411    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:14.807606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:14.807476    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:15.362962 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:15.362896    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:15.763597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:15.760858    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 10:45:15.763597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:15.760928    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 10:45:15.818617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:15.807269    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:15.818617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:15.807343    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:15.906648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:15.895775    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:16.076201 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.076156    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-xks-ca\") pod \"948d9d84-85d4-4a10-9873-674500d7ff77\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") "
Nov 03 10:45:16.092614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.086693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:16.103617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.093010    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-openshift-ca\") pod \"948d9d84-85d4-4a10-9873-674500d7ff77\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") "
Nov 03 10:45:16.121306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.113232    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948d9d84-85d4-4a10-9873-674500d7ff77-kube-api-access-2xntc\" (UniqueName: \"kubernetes.io/projected/948d9d84-85d4-4a10-9873-674500d7ff77-kube-api-access-2xntc\") pod \"948d9d84-85d4-4a10-9873-674500d7ff77\" (UID: \"948d9d84-85d4-4a10-9873-674500d7ff77\") "
Nov 03 10:45:16.133629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.107817    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-openshift-ca" pod "948d9d84-85d4-4a10-9873-674500d7ff77" (UID: "948d9d84-85d4-4a10-9873-674500d7ff77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:16.154598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.142486    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:16.210600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.186114    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-xks-ca" pod "948d9d84-85d4-4a10-9873-674500d7ff77" (UID: "948d9d84-85d4-4a10-9873-674500d7ff77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:16.210600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.186665    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948d9d84-85d4-4a10-9873-674500d7ff77-kube-api-access-2xntc" pod "948d9d84-85d4-4a10-9873-674500d7ff77" (UID: "948d9d84-85d4-4a10-9873-674500d7ff77"). InnerVolumeSpecName "kube-api-access-2xntc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:16.284766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.284737    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xntc\" (UniqueName: \"kubernetes.io/projected/948d9d84-85d4-4a10-9873-674500d7ff77-kube-api-access-2xntc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:16.284766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.284763    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948d9d84-85d4-4a10-9873-674500d7ff77-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:16.470705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.470620    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6" event={"ID":"948d9d84-85d4-4a10-9873-674500d7ff77","Type":"ContainerDied","Data":"7412898220d42bc7d5c65dc7237e8935a192559ab3b684c3db2da514aff04751"}
Nov 03 10:45:16.470705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.470691    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7412898220d42bc7d5c65dc7237e8935a192559ab3b684c3db2da514aff04751"
Nov 03 10:45:16.470705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.470630    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"
Nov 03 10:45:16.755662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.755163    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 10:45:16.755662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.755221    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 10:45:16.816598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.808077    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:16.816598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.808136    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:16.931076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:16.931044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:17.753607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:17.753276    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 10:45:17.753607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:17.753347    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 10:45:17.803952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:17.803915    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:17.804238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:17.804211    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:17.859800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:17.859440    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:18.160276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.156901    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"]
Nov 03 10:45:18.160276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.157940    5817 cpu_manager.go:414] "containerMap: removing container" podUID="948d9d84-85d4-4a10-9873-674500d7ff77" containerName="revoke-keys"
Nov 03 10:45:18.160276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.157960    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948d9d84-85d4-4a10-9873-674500d7ff77" containerName="revoke-keys"
Nov 03 10:45:18.160276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.158169    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="948d9d84-85d4-4a10-9873-674500d7ff77" containerName="revoke-keys"
Nov 03 10:45:18.162799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.162770    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.164136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.164099    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"]
Nov 03 10:45:18.166545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.166519    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:18.271481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.271263    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.271481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.271308    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd46w\" (UniqueName: \"kubernetes.io/projected/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-kube-api-access-cd46w\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.271481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.271329    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.373244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.373215    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.373416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.373251    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd46w\" (UniqueName: \"kubernetes.io/projected/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-kube-api-access-cd46w\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.373416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.373274    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.374120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.374097    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.378607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.375948    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.382219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.382199    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd46w\" (UniqueName: \"kubernetes.io/projected/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-kube-api-access-cd46w\") pod \"maas-api-revoke-sub-2396fbd2-pfmns\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.481824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.481796    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:18.612598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.612241    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:18.758604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.758482    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 10:45:18.759265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.759174    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 10:45:18.804904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.804477    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:18.804904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.804532    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:18.960628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:18.958833    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"]
Nov 03 10:45:18.964231 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:18.964207    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a2c7c0_c05f_4a5b_bc6b_b8c81a302422.slice/crio-6e7dd0f42cca870ab52a7c91ff792f1c51ee9b587879063d18b581f4b8f84ab4 WatchSource:0}: Error finding container 6e7dd0f42cca870ab52a7c91ff792f1c51ee9b587879063d18b581f4b8f84ab4: Status 404 returned error can't find the container with id 6e7dd0f42cca870ab52a7c91ff792f1c51ee9b587879063d18b581f4b8f84ab4
Nov 03 10:45:19.406246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:19.406206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:19.606823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:19.603091    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns" event={"ID":"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422","Type":"ContainerStarted","Data":"6e7dd0f42cca870ab52a7c91ff792f1c51ee9b587879063d18b581f4b8f84ab4"}
Nov 03 10:45:19.747348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:19.746930    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 10:45:19.747348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:19.746988    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 10:45:19.810596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:19.809950    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:19.810596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:19.810011    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:20.768686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:20.768649    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 10:45:20.791067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:20.773672    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 10:45:20.791401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:20.791368    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:20.792245 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:20.792217    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:20.813187 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:20.805913    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:20.813187 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:20.805958    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:21.049659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:20.965983    5817 scope.go:122] "RemoveContainer" containerID="2241a44b21deba3ca53b53f7fe7c7b718d328f4c5f9acda9fefbd7bf14702d52"
Nov 03 10:45:21.049659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:20.967185    5817 scope.go:122] "RemoveContainer" containerID="8537008c6e2eaf24a6aa2faf3d1ddf47a6eeab1c38f787bfaa01a1af1cb6f353"
Nov 03 10:45:21.120694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.119210    5817 generic.go:356] "Generic (PLEG): container finished" podID="38a2c7c0-c05f-4a5b-bc6b-b8c81a302422" containerID="a9ea7d21ee16ab5630cb353e6bc1be5ce87e86a8dc38bfa7800a61f094169f40" exitCode=0
Nov 03 10:45:21.120694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.119401    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns" event={"ID":"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422","Type":"ContainerDied","Data":"a9ea7d21ee16ab5630cb353e6bc1be5ce87e86a8dc38bfa7800a61f094169f40"}
Nov 03 10:45:21.168535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.167666    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:21.252781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.252738    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:21.287945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.257885    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:45:21.899657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.801697    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:21.899657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.802434    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:45:21.899657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.804267    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:21.899657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.804304    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:21.899657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.804357    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:21.899657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:21.824969    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:45:22.144137 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.144097    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.177960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.176140    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.177960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.176790    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:45:22.289363 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.289315    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.289916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.289884    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.307726 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.307668    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:22.335720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.308402    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:45:22.356618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.355695    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.356874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.310969    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.375581 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.375453    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.401872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.401826    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:45:22.830814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.810649    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:22.830814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.812218    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:45:22.830814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.815223    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:45:22.830814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.815267    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 10:45:22.892927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.888116    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:22.892927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.888183    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.901601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.893269    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.901601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.897196    5817 scope.go:122] "RemoveContainer" containerID="3d24754f6cb2c19f9369380b23aad2af00c4ae754785b6f5b9c1b3fa6c11ba00"
Nov 03 10:45:22.902308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:22.902225    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.213354 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.173433    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:23.213354 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.187695    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.272356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.272303    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.306723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.280205    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:45:23.306723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.301942    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.367650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.355285    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:23.367650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.359382    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.457899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.403734    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:45:23.457899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.403950    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:45:23.457899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.426035    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:45:23.457899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.427706    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:45:23.457899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.432652    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:45:23.945600 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:23.935403    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:45:23.949701 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.949660    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:23.954001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.953967    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:45:23.957939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.957908    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.961958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:23.961935    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:45:24.062917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.004708    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:24.062917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.004772    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.088599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.084842    5817 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:24.162660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.124684    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.378598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.362283    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.378598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.364083    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.378598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.364404    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:45:24.404882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.273552    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:45:24.468629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.450720    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.468629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.451148    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:45:24.644487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.637681    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.862674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.825822    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:24.960807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.960768    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.961369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.922482    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:24.961503 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.961480    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:25.063696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.973269    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:25.149633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.979481    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:25.149633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.985400    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:25.149633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:24.987167    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded"
Nov 03 10:45:25.149633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.012659    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:25.149633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.147552    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:25.203601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.178208    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:25.501647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.494690    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:25.501647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.396143    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:25.501647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.396106    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:45:25.501647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.396161    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:45:25.522098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.442686    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:25.522098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.467908    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:25.522098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.467943    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:45:25.726624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:25.639118    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:26.066749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.051815    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:26.066749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.051905    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:26.066749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.053636    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:26.069478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.068696    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:26.069478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.068764    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:26.152036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.151972    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:26.252928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.156636    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:45:26.252928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.156944    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:26.252928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.248678    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:26.252928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.248739    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:26.252928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.252842    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Nov 03 10:45:26.256608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.254670    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:26.415874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.400171    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-openshift-ca\") pod \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") "
Nov 03 10:45:26.415874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.400338    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-xks-ca\") pod \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") "
Nov 03 10:45:26.415874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.400455    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-kube-api-access-cd46w\" (UniqueName: \"kubernetes.io/projected/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-kube-api-access-cd46w\") pod \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\" (UID: \"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422\") "
Nov 03 10:45:26.437137 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.437097    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-openshift-ca" pod "38a2c7c0-c05f-4a5b-bc6b-b8c81a302422" (UID: "38a2c7c0-c05f-4a5b-bc6b-b8c81a302422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:26.446246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.440621    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-xks-ca" pod "38a2c7c0-c05f-4a5b-bc6b-b8c81a302422" (UID: "38a2c7c0-c05f-4a5b-bc6b-b8c81a302422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:26.499695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.499658    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:26.520609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.511867    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-kube-api-access-cd46w" pod "38a2c7c0-c05f-4a5b-bc6b-b8c81a302422" (UID: "38a2c7c0-c05f-4a5b-bc6b-b8c81a302422"). InnerVolumeSpecName "kube-api-access-cd46w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:26.520609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.511924    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:26.520609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.512202    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:45:26.520609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.513890    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:45:26.523936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.523914    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:45:26.526671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.525619    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:45:26.538873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.538855    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:26.539034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.539020    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd46w\" (UniqueName: \"kubernetes.io/projected/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-kube-api-access-cd46w\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:26.545141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.545124    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:26.649565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.649471    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:45:26.702449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.701552    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:45:26.704589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.704397    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:45:26.713605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.706074    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:45:26.713605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.706239    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:45:26.713605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.710239    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:45:26.713605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.712539    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:45:26.735592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.727397    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:45:26.738305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.738245    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:45:26.762898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.762873    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:45:26.879704 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.879479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:26.915658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.915345    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns" event={"ID":"38a2c7c0-c05f-4a5b-bc6b-b8c81a302422","Type":"ContainerDied","Data":"6e7dd0f42cca870ab52a7c91ff792f1c51ee9b587879063d18b581f4b8f84ab4"}
Nov 03 10:45:26.915658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.915445    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e7dd0f42cca870ab52a7c91ff792f1c51ee9b587879063d18b581f4b8f84ab4"
Nov 03 10:45:26.915658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.915604    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"
Nov 03 10:45:26.937330 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:26.937298    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" event={"ID":"a2f6acfc-e13b-4eb9-ab55-71d8242cde12","Type":"ContainerStarted","Data":"ebf11c145e18f866fc422a1c9c84ac1b6c3c5526f4575f04bf40eb6b5e5816ad"}
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.024897    5817 generic.go:356] "Generic (PLEG): container finished" podID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerID="2ffc2a4c6bb0969c64c484c6d86a47f244ec529de2e7d97bd80172f5211f68f6" exitCode=60
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.025078    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" event={"ID":"6eb8a820-25cb-4122-bc8b-3968b7252e34","Type":"ContainerDied","Data":"2ffc2a4c6bb0969c64c484c6d86a47f244ec529de2e7d97bd80172f5211f68f6"}
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.025133    5817 scope.go:122] "RemoveContainer" containerID="2241a44b21deba3ca53b53f7fe7c7b718d328f4c5f9acda9fefbd7bf14702d52"
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.025687    5817 scope.go:122] "RemoveContainer" containerID="2ffc2a4c6bb0969c64c484c6d86a47f244ec529de2e7d97bd80172f5211f68f6"
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:27.026078    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l_odh-ai-gateway-infra(6eb8a820-25cb-4122-bc8b-3968b7252e34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34"
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.040308    5817 generic.go:356] "Generic (PLEG): container finished" podID="19932057-de3d-45b1-b4ea-a64c2019943d" containerID="d0471bca3946b1384140ad56c50c68afe9135fb53b9ab76d82701e6a7b86ed21" exitCode=60
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.041349    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" event={"ID":"19932057-de3d-45b1-b4ea-a64c2019943d","Type":"ContainerDied","Data":"d0471bca3946b1384140ad56c50c68afe9135fb53b9ab76d82701e6a7b86ed21"}
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.042726    5817 scope.go:122] "RemoveContainer" containerID="d0471bca3946b1384140ad56c50c68afe9135fb53b9ab76d82701e6a7b86ed21"
Nov 03 10:45:27.049597 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:27.043076    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z_odh-ai-gateway-infra(19932057-de3d-45b1-b4ea-a64c2019943d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" podUID="19932057-de3d-45b1-b4ea-a64c2019943d"
Nov 03 10:45:27.146123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.145754    5817 scope.go:122] "RemoveContainer" containerID="8537008c6e2eaf24a6aa2faf3d1ddf47a6eeab1c38f787bfaa01a1af1cb6f353"
Nov 03 10:45:27.151498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.151441    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:27.153208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.153158    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:27.153916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.153872    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:45:27.163033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.163007    5817 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:27.163116 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.163062    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:27.708044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.708008    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:27.708557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.708310    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:27.709037 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.709016    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 10:45:27.837253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:27.837221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:28.010796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.010715    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"]
Nov 03 10:45:28.054164 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.054134    5817 generic.go:356] "Generic (PLEG): container finished" podID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerID="ebf11c145e18f866fc422a1c9c84ac1b6c3c5526f4575f04bf40eb6b5e5816ad" exitCode=60
Nov 03 10:45:28.054347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.054216    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" event={"ID":"a2f6acfc-e13b-4eb9-ab55-71d8242cde12","Type":"ContainerDied","Data":"ebf11c145e18f866fc422a1c9c84ac1b6c3c5526f4575f04bf40eb6b5e5816ad"}
Nov 03 10:45:28.054347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.054284    5817 scope.go:122] "RemoveContainer" containerID="3d24754f6cb2c19f9369380b23aad2af00c4ae754785b6f5b9c1b3fa6c11ba00"
Nov 03 10:45:28.059898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.059868    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"]
Nov 03 10:45:28.085639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.085610    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"]
Nov 03 10:45:28.378298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.378262    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:28.381067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.381049    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:28.384760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.384742    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:28.528448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528417    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19932057-de3d-45b1-b4ea-a64c2019943d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19932057-de3d-45b1-b4ea-a64c2019943d-maas-api-service-ca\") pod \"19932057-de3d-45b1-b4ea-a64c2019943d\" (UID: \"19932057-de3d-45b1-b4ea-a64c2019943d\") "
Nov 03 10:45:28.528627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528538    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb8a820-25cb-4122-bc8b-3968b7252e34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a820-25cb-4122-bc8b-3968b7252e34-maas-api-service-ca\") pod \"6eb8a820-25cb-4122-bc8b-3968b7252e34\" (UID: \"6eb8a820-25cb-4122-bc8b-3968b7252e34\") "
Nov 03 10:45:28.528696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528649    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-maas-api-service-ca\") pod \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\" (UID: \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\") "
Nov 03 10:45:28.528756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528716    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-kube-api-access-v6h5f\" (UniqueName: \"kubernetes.io/projected/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-kube-api-access-v6h5f\") pod \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\" (UID: \"a2f6acfc-e13b-4eb9-ab55-71d8242cde12\") "
Nov 03 10:45:28.528823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528804    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19932057-de3d-45b1-b4ea-a64c2019943d-kube-api-access-q2kvr\" (UniqueName: \"kubernetes.io/projected/19932057-de3d-45b1-b4ea-a64c2019943d-kube-api-access-q2kvr\") pod \"19932057-de3d-45b1-b4ea-a64c2019943d\" (UID: \"19932057-de3d-45b1-b4ea-a64c2019943d\") "
Nov 03 10:45:28.528912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528887    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb8a820-25cb-4122-bc8b-3968b7252e34-maas-api-service-ca" pod "6eb8a820-25cb-4122-bc8b-3968b7252e34" (UID: "6eb8a820-25cb-4122-bc8b-3968b7252e34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:28.528977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528863    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19932057-de3d-45b1-b4ea-a64c2019943d-maas-api-service-ca" pod "19932057-de3d-45b1-b4ea-a64c2019943d" (UID: "19932057-de3d-45b1-b4ea-a64c2019943d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:28.528977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528970    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb8a820-25cb-4122-bc8b-3968b7252e34-kube-api-access-b2457\" (UniqueName: \"kubernetes.io/projected/6eb8a820-25cb-4122-bc8b-3968b7252e34-kube-api-access-b2457\") pod \"6eb8a820-25cb-4122-bc8b-3968b7252e34\" (UID: \"6eb8a820-25cb-4122-bc8b-3968b7252e34\") "
Nov 03 10:45:28.529092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.528994    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-maas-api-service-ca" pod "a2f6acfc-e13b-4eb9-ab55-71d8242cde12" (UID: "a2f6acfc-e13b-4eb9-ab55-71d8242cde12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:28.529638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.529616    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19932057-de3d-45b1-b4ea-a64c2019943d-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.529807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.529641    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a820-25cb-4122-bc8b-3968b7252e34-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.529807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.529656    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.531615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.531547    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19932057-de3d-45b1-b4ea-a64c2019943d-kube-api-access-q2kvr" pod "19932057-de3d-45b1-b4ea-a64c2019943d" (UID: "19932057-de3d-45b1-b4ea-a64c2019943d"). InnerVolumeSpecName "kube-api-access-q2kvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:28.531823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.531802    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-kube-api-access-v6h5f" pod "a2f6acfc-e13b-4eb9-ab55-71d8242cde12" (UID: "a2f6acfc-e13b-4eb9-ab55-71d8242cde12"). InnerVolumeSpecName "kube-api-access-v6h5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:28.531892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.531876    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb8a820-25cb-4122-bc8b-3968b7252e34-kube-api-access-b2457" pod "6eb8a820-25cb-4122-bc8b-3968b7252e34" (UID: "6eb8a820-25cb-4122-bc8b-3968b7252e34"). InnerVolumeSpecName "kube-api-access-b2457". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:28.631197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.631128    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2457\" (UniqueName: \"kubernetes.io/projected/6eb8a820-25cb-4122-bc8b-3968b7252e34-kube-api-access-b2457\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.631197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.631152    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6h5f\" (UniqueName: \"kubernetes.io/projected/a2f6acfc-e13b-4eb9-ab55-71d8242cde12-kube-api-access-v6h5f\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.631197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.631163    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2kvr\" (UniqueName: \"kubernetes.io/projected/19932057-de3d-45b1-b4ea-a64c2019943d-kube-api-access-q2kvr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.711946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:28.711908    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:29.074201 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.074169    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"
Nov 03 10:45:29.074406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.074221    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv" event={"ID":"a2f6acfc-e13b-4eb9-ab55-71d8242cde12","Type":"ContainerDied","Data":"f707f7fddeb0e71371765f9bb6841c8a4e476994104eae832f116fba77438693"}
Nov 03 10:45:29.074406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.074284    5817 scope.go:122] "RemoveContainer" containerID="ebf11c145e18f866fc422a1c9c84ac1b6c3c5526f4575f04bf40eb6b5e5816ad"
Nov 03 10:45:29.075952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.075932    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"
Nov 03 10:45:29.076099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.075945    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l" event={"ID":"6eb8a820-25cb-4122-bc8b-3968b7252e34","Type":"ContainerDied","Data":"df4839c9d7c78cd29bb86a4fdb4757179ca7147eb804d318f9d7bca4eaa2cd28"}
Nov 03 10:45:29.077700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.077679    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z" event={"ID":"19932057-de3d-45b1-b4ea-a64c2019943d","Type":"ContainerDied","Data":"ce2062d1f423438053f586ef50999746850a8f618b65252b9d27418d290dc44d"}
Nov 03 10:45:29.077866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.077851    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"
Nov 03 10:45:29.088278 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.088242    5817 scope.go:122] "RemoveContainer" containerID="2ffc2a4c6bb0969c64c484c6d86a47f244ec529de2e7d97bd80172f5211f68f6"
Nov 03 10:45:29.099026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.099007    5817 scope.go:122] "RemoveContainer" containerID="d0471bca3946b1384140ad56c50c68afe9135fb53b9ab76d82701e6a7b86ed21"
Nov 03 10:45:29.111199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.109474    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"]
Nov 03 10:45:29.114135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.114113    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895045-j9pjv"]
Nov 03 10:45:29.127133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.127107    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"]
Nov 03 10:45:29.131423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.131403    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895045-87p8z"]
Nov 03 10:45:29.142774 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.142753    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"]
Nov 03 10:45:29.149464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.149445    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895045-4w25l"]
Nov 03 10:45:29.503025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.502993    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:29.765323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.765245    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:29.910811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.910770    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" path="/var/lib/kubelet/pods/19932057-de3d-45b1-b4ea-a64c2019943d/volumes"
Nov 03 10:45:29.911486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.911460    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" path="/var/lib/kubelet/pods/6eb8a820-25cb-4122-bc8b-3968b7252e34/volumes"
Nov 03 10:45:29.912217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:29.912188    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" path="/var/lib/kubelet/pods/a2f6acfc-e13b-4eb9-ab55-71d8242cde12/volumes"
Nov 03 10:45:30.089536 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.089455    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"]
Nov 03 10:45:30.090182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090157    5817 cpu_manager.go:414] "containerMap: removing container" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:30.090182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090182    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090204    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090209    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090221    5817 cpu_manager.go:414] "containerMap: removing container" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090226    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090238    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090243    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090251    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090256    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090263    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090267    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090276    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090280    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090304    5817 cpu_manager.go:414] "containerMap: removing container" podUID="38a2c7c0-c05f-4a5b-bc6b-b8c81a302422" containerName="revoke-keys"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090309    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a2c7c0-c05f-4a5b-bc6b-b8c81a302422" containerName="revoke-keys"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090319    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.090394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090323    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090440    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090452    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090462    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090469    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090478    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090487    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb8a820-25cb-4122-bc8b-3968b7252e34" containerName="cleanup"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090503    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090514    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a2c7c0-c05f-4a5b-bc6b-b8c81a302422" containerName="revoke-keys"
Nov 03 10:45:30.091298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.090524    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f6acfc-e13b-4eb9-ab55-71d8242cde12" containerName="cleanup"
Nov 03 10:45:30.096448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.096425    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.106347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.106329    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ab592834-openshift-default-dockercfg-vq7pv\""
Nov 03 10:45:30.117672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.117645    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"]
Nov 03 10:45:30.124084 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.124059    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:45:30.190507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.190477    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:30.254861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.254834    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.254970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.254902    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbhw\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-kube-api-access-4pbhw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.255066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.255002    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.255066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.255031    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.255066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.255057    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.255250 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.255192    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.255250 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.255213    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0086c91-894b-4b31-8eb7-7c147e4512ba-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.255361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.255249    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.255432 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.255398    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.342163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.340850    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:30.362591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.362532    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbhw\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-kube-api-access-4pbhw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.362784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.362639    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.362784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.362667    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.362784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.362700    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.362955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.362805    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.362955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.362830    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0086c91-894b-4b31-8eb7-7c147e4512ba-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.362955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.362874    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.363119 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.362968    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.363119 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.363070    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.363560 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.363535    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.366437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.366408    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0086c91-894b-4b31-8eb7-7c147e4512ba-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.376657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.366818    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.376657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.367040    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.376657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.368962    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.386539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.385718    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.388629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.388542    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.388793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.388768    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.389145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.389124    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbhw\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-kube-api-access-4pbhw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.421818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.421785    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:30.675703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.672481    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:30.750700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.750673    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"]
Nov 03 10:45:30.751474 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:30.751450    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0086c91_894b_4b31_8eb7_7c147e4512ba.slice/crio-e5085f23e849be568e10e254e11619a3481e4b1a68e8c9e37d4721d45d07f6ad WatchSource:0}: Error finding container e5085f23e849be568e10e254e11619a3481e4b1a68e8c9e37d4721d45d07f6ad: Status 404 returned error can't find the container with id e5085f23e849be568e10e254e11619a3481e4b1a68e8c9e37d4721d45d07f6ad
Nov 03 10:45:30.754295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.754266    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:30.754396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.754334    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:30.754396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:30.754361    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:31.109272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:31.109230    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" event={"ID":"d0086c91-894b-4b31-8eb7-7c147e4512ba","Type":"ContainerStarted","Data":"80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0"}
Nov 03 10:45:31.109467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:31.109286    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" event={"ID":"d0086c91-894b-4b31-8eb7-7c147e4512ba","Type":"ContainerStarted","Data":"e5085f23e849be568e10e254e11619a3481e4b1a68e8c9e37d4721d45d07f6ad"}
Nov 03 10:45:31.129106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:31.129050    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podStartSLOduration=1.129031442 podStartE2EDuration="1.129031442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:31.128033153 +0000 UTC m=+961.333615205" watchObservedRunningTime="2026-11-03 10:45:31.129031442 +0000 UTC m=+961.334613494"
Nov 03 10:45:31.433875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:31.425643    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:31.433875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:31.425794    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 10:45:31.433875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:31.425849    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 10:45:32.437482 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:32.437445    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:33.267465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:33.267434    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:33.270848 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:33.270817    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:45:33.287961 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:33.285063    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:33.876587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:33.876542    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:34.046869 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.046829    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"]
Nov 03 10:45:34.047526 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.047504    5817 cpu_manager.go:414] "containerMap: removing container" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:34.047669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.047528    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:34.047744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.047683    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="19932057-de3d-45b1-b4ea-a64c2019943d" containerName="cleanup"
Nov 03 10:45:34.051506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.051485    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.068782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.068756    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:34.070085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.070064    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:45:34.071154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.071134    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"]
Nov 03 10:45:34.119210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.119175    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:34.146227 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.146171    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.146320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.146239    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqnjb\" (UniqueName: \"kubernetes.io/projected/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-kube-api-access-kqnjb\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.146370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.146342    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.248345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.248310    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.248530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.248494    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqnjb\" (UniqueName: \"kubernetes.io/projected/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-kube-api-access-kqnjb\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.248628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.248597    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.249179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.249145    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.251345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.251324    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.256897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.256876    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqnjb\" (UniqueName: \"kubernetes.io/projected/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-kube-api-access-kqnjb\") pod \"maas-api-revoke-sub-3fa4a73b-qgwkv\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.365991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.365960    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:34.656826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.656797    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"]
Nov 03 10:45:34.656937 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:34.656855    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b2ec74_dfdb_469e_9748_a8c0c43b1362.slice/crio-4339fb8efbdf7690afc564630041db14ba7f8c84ba2523c816a022fb3f02f7cf WatchSource:0}: Error finding container 4339fb8efbdf7690afc564630041db14ba7f8c84ba2523c816a022fb3f02f7cf: Status 404 returned error can't find the container with id 4339fb8efbdf7690afc564630041db14ba7f8c84ba2523c816a022fb3f02f7cf
Nov 03 10:45:34.974781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:34.973863    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:35.183544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:35.183509    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:35.293322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:35.293229    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv" event={"ID":"f7b2ec74-dfdb-469e-9748-a8c0c43b1362","Type":"ContainerStarted","Data":"4eab510d32c1262527dd6690989db7a2be1571efa04a0d4d66490eb9f85d21a4"}
Nov 03 10:45:35.293322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:35.293289    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv" event={"ID":"f7b2ec74-dfdb-469e-9748-a8c0c43b1362","Type":"ContainerStarted","Data":"4339fb8efbdf7690afc564630041db14ba7f8c84ba2523c816a022fb3f02f7cf"}
Nov 03 10:45:35.314611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:35.314524    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv" podStartSLOduration=1.314506921 podStartE2EDuration="1.314506921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:35.308733352 +0000 UTC m=+965.514315414" watchObservedRunningTime="2026-11-03 10:45:35.314506921 +0000 UTC m=+965.520088973"
Nov 03 10:45:35.319993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:35.319963    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:35.860257 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:35.860152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:36.003649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:36.003616    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:36.309637 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:36.309606    5817 generic.go:356] "Generic (PLEG): container finished" podID="f7b2ec74-dfdb-469e-9748-a8c0c43b1362" containerID="4eab510d32c1262527dd6690989db7a2be1571efa04a0d4d66490eb9f85d21a4" exitCode=0
Nov 03 10:45:36.309849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:36.309627    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv" event={"ID":"f7b2ec74-dfdb-469e-9748-a8c0c43b1362","Type":"ContainerDied","Data":"4eab510d32c1262527dd6690989db7a2be1571efa04a0d4d66490eb9f85d21a4"}
Nov 03 10:45:36.371133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:36.371085    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:36.515779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:36.515748    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:37.051733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.051364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:37.516397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.516371    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:37.624054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.623970    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-openshift-ca\") pod \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") "
Nov 03 10:45:37.624206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.624164    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-kube-api-access-kqnjb\" (UniqueName: \"kubernetes.io/projected/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-kube-api-access-kqnjb\") pod \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") "
Nov 03 10:45:37.624296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.624269    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-xks-ca\") pod \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\" (UID: \"f7b2ec74-dfdb-469e-9748-a8c0c43b1362\") "
Nov 03 10:45:37.624359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.624335    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-openshift-ca" pod "f7b2ec74-dfdb-469e-9748-a8c0c43b1362" (UID: "f7b2ec74-dfdb-469e-9748-a8c0c43b1362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:37.624829 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.624798    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:37.626449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.626420    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-xks-ca" pod "f7b2ec74-dfdb-469e-9748-a8c0c43b1362" (UID: "f7b2ec74-dfdb-469e-9748-a8c0c43b1362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:37.626554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.626458    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-kube-api-access-kqnjb" pod "f7b2ec74-dfdb-469e-9748-a8c0c43b1362" (UID: "f7b2ec74-dfdb-469e-9748-a8c0c43b1362"). InnerVolumeSpecName "kube-api-access-kqnjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:37.736594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.735973    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqnjb\" (UniqueName: \"kubernetes.io/projected/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-kube-api-access-kqnjb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:37.736594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.736008    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b2ec74-dfdb-469e-9748-a8c0c43b1362-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:37.861476 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:37.861447    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:38.343892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:38.343855    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv" event={"ID":"f7b2ec74-dfdb-469e-9748-a8c0c43b1362","Type":"ContainerDied","Data":"4339fb8efbdf7690afc564630041db14ba7f8c84ba2523c816a022fb3f02f7cf"}
Nov 03 10:45:38.344493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:38.344474    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4339fb8efbdf7690afc564630041db14ba7f8c84ba2523c816a022fb3f02f7cf"
Nov 03 10:45:38.344493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:38.344423    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"
Nov 03 10:45:38.749547 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:38.747101    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:38.885530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:38.885494    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:39.749544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:39.749516    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:39.888159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:39.888125    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:40.030317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:40.030227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:40.639735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:40.639701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:40.789332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:40.789301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:40.938588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:40.938541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:41.377425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:41.377338    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:42.039869 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.039832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:42.450595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.443299    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:42.640992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.625793    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"]
Nov 03 10:45:42.640992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.626801    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f7b2ec74-dfdb-469e-9748-a8c0c43b1362" containerName="revoke-keys"
Nov 03 10:45:42.640992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.626824    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b2ec74-dfdb-469e-9748-a8c0c43b1362" containerName="revoke-keys"
Nov 03 10:45:42.640992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.627014    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b2ec74-dfdb-469e-9748-a8c0c43b1362" containerName="revoke-keys"
Nov 03 10:45:42.665889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.665850    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:42.706653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.668384    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:42.706653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.702425    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-141053\""
Nov 03 10:45:42.706653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.702492    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-141053-dockercfg-z7krl\""
Nov 03 10:45:42.756597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.732039    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"]
Nov 03 10:45:42.781596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.757989    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5bh\" (UniqueName: \"kubernetes.io/projected/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-kube-api-access-hp5bh\") pod \"payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk\" (UID: \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:42.781596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.758119    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk\" (UID: \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:42.805596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.781887    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"]
Nov 03 10:45:42.977991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.928485    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5bh\" (UniqueName: \"kubernetes.io/projected/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-kube-api-access-hp5bh\") pod \"payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk\" (UID: \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:42.993600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.978375    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk\" (UID: \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:42.993600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:42.980199    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk\" (UID: \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:43.173095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.002281    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"]
Nov 03 10:45:43.173095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.002398    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:43.173095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.036266    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5bh\" (UniqueName: \"kubernetes.io/projected/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-kube-api-access-hp5bh\") pod \"payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk\" (UID: \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:43.193597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.174830    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdxh\" (UniqueName: \"kubernetes.io/projected/29c3c4c4-6087-4653-9e86-8ce491859398-kube-api-access-gxdxh\") pod \"payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2\" (UID: \"29c3c4c4-6087-4653-9e86-8ce491859398\") " pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:43.193597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.175036    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c3c4c4-6087-4653-9e86-8ce491859398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2\" (UID: \"29c3c4c4-6087-4653-9e86-8ce491859398\") " pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:43.202929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.202897    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:43.229175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.225161    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:45:43.308598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.283452    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c3c4c4-6087-4653-9e86-8ce491859398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2\" (UID: \"29c3c4c4-6087-4653-9e86-8ce491859398\") " pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:43.309102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.309069    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdxh\" (UniqueName: \"kubernetes.io/projected/29c3c4c4-6087-4653-9e86-8ce491859398-kube-api-access-gxdxh\") pod \"payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2\" (UID: \"29c3c4c4-6087-4653-9e86-8ce491859398\") " pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:43.309811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.295527    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c3c4c4-6087-4653-9e86-8ce491859398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2\" (UID: \"29c3c4c4-6087-4653-9e86-8ce491859398\") " pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:43.349922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.336158    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdxh\" (UniqueName: \"kubernetes.io/projected/29c3c4c4-6087-4653-9e86-8ce491859398-kube-api-access-gxdxh\") pod \"payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2\" (UID: \"29c3c4c4-6087-4653-9e86-8ce491859398\") " pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:43.384621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.371028    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:43.444050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.441661    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.444050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.443007    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.483962 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.460453    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:45:43.535620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.515211    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:45:43.535620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.534378    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.666021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.665956    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.735757 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.729692    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:45:43.820711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.818662    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.823260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:43.822932    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.242598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.239207    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.254597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.251171    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:45:44.478630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.470763    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.478630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.471597    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:45:44.576655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.555176    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.602598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.564674    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.603188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.564757    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.603655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.564855    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.607591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.600929    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:45:44.635594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.611714    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:45:44.635924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.635902    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:45:44.636116 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.636103    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:45:44.837649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.837375    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.854737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.838427    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:45:45.023088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.924181    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:45.023088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.925690    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.023088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.926150    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:45:45.023088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.933842    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.023088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.934961    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:45:45.023088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.936283    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.023088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:44.937129    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:45:45.282883 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:45.282842    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:45.484715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:45.484672    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:45.636825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:45.636774    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.637852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:45.637819    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.639376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:45.639352    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.640401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:45.640377    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.022993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.022935    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.119558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.119507    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2xhzm" podUID="042afe48-f39e-4b28-bb65-d7db0e147638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.120987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.120958    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.161663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.161612    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.231881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.179808    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.282399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.265897    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.336338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.284281    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.342770 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.342737    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:46.342978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.342953    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.487856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.487784    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.698931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.667028    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.700923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.700861    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.722046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.702500    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.722046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.702743    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.927594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.866344    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.927594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:46.898915    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:47.023807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.023760    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:47.139592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.129529    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:45:47.202072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.184276    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:45:47.202072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.194666    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:45:47.524408 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:47.524365    5817 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.631s"
Nov 03 10:45:47.526422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.524995    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:45:47.526422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.525036    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"]
Nov 03 10:45:47.563595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.559702    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:47.583006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.577718    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:45:47.583006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.577795    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:45:47.583006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.577816    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"]
Nov 03 10:45:47.583006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.577860    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:45:47.583006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.577891    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:45:47.583006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.577971    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.583006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.578767    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:47.639607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.588179    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:45:47.644340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.644314    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:45:47.662222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.651976    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:47.662222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.653099    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:45:47.664231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.664183    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:45:47.664399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.664388    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:45:47.667100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.667077    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk" event={"ID":"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853","Type":"ContainerStarted","Data":"25e460d8788a1b58e153f463b80743c66ba8849d3af3d2e8a8f6ecd67faae39f"}
Nov 03 10:45:47.670212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.670193    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2" event={"ID":"29c3c4c4-6087-4653-9e86-8ce491859398","Type":"ContainerStarted","Data":"a90d434f22a11f42dcfb43e27f605a9068da8f4facdff86600d5b82834074f0e"}
Nov 03 10:45:47.670748 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.670725    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:47.672809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.672759    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:45:47.736612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.724749    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxtmw\" (UniqueName: \"kubernetes.io/projected/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-kube-api-access-zxtmw\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.736612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.724939    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.736612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.725147    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.752179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.752152    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"]
Nov 03 10:45:47.790535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.790490    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"]
Nov 03 10:45:47.827153 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.827103    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxtmw\" (UniqueName: \"kubernetes.io/projected/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-kube-api-access-zxtmw\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.827737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.827711    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.828424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.828402    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.829335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.829312    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.830939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.830914    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.833023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.832148    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2" podStartSLOduration=5.832137119 podStartE2EDuration="5.832137119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:47.818975707 +0000 UTC m=+978.024557758" watchObservedRunningTime="2026-11-03 10:45:47.832137119 +0000 UTC m=+978.037719185"
Nov 03 10:45:47.835440 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.835413    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxtmw\" (UniqueName: \"kubernetes.io/projected/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-kube-api-access-zxtmw\") pod \"maas-api-revoke-sub-77cb09d0-nbd9v\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:47.967422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:47.967395    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:48.182303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.182147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:48.292326 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:48.292295    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0983ec40_cd53_44fe_bfbf_7a42f00e52f6.slice/crio-35370a032d2e09a5dea7bff5960d18407bd3f05ee762c70c6ee9815cbaee6f7c WatchSource:0}: Error finding container 35370a032d2e09a5dea7bff5960d18407bd3f05ee762c70c6ee9815cbaee6f7c: Status 404 returned error can't find the container with id 35370a032d2e09a5dea7bff5960d18407bd3f05ee762c70c6ee9815cbaee6f7c
Nov 03 10:45:48.294616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.294591    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"]
Nov 03 10:45:48.680273 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.680241    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v" event={"ID":"0983ec40-cd53-44fe-bfbf-7a42f00e52f6","Type":"ContainerStarted","Data":"179bded08410c50380cce5e19b42f9f2367c249125984a7a158e9fcb0c1df110"}
Nov 03 10:45:48.680701 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.680300    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v" event={"ID":"0983ec40-cd53-44fe-bfbf-7a42f00e52f6","Type":"ContainerStarted","Data":"35370a032d2e09a5dea7bff5960d18407bd3f05ee762c70c6ee9815cbaee6f7c"}
Nov 03 10:45:48.682077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.682036    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk" event={"ID":"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853","Type":"ContainerStarted","Data":"4503f8f2bd25ec34382c8cbbdde5ed0205fb54dcb1d64bb1ea714caab018e11c"}
Nov 03 10:45:48.682155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.682123    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:48.683688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.683663    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2" event={"ID":"29c3c4c4-6087-4653-9e86-8ce491859398","Type":"ContainerStarted","Data":"3a12cbcf8dc27588729020f07c178d9e60b2c9d062b394280af23b40468c7757"}
Nov 03 10:45:48.698014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.697970    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v" podStartSLOduration=4.697959029 podStartE2EDuration="4.697959029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:48.696381751 +0000 UTC m=+978.901963803" watchObservedRunningTime="2026-11-03 10:45:48.697959029 +0000 UTC m=+978.903541079"
Nov 03 10:45:48.715491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:48.715441    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk" podStartSLOduration=6.715423834 podStartE2EDuration="6.715423834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:48.712466107 +0000 UTC m=+978.918048158" watchObservedRunningTime="2026-11-03 10:45:48.715423834 +0000 UTC m=+978.921005936"
Nov 03 10:45:49.099417 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:49.098943    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:49.702971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:49.702940    5817 generic.go:356] "Generic (PLEG): container finished" podID="0983ec40-cd53-44fe-bfbf-7a42f00e52f6" containerID="179bded08410c50380cce5e19b42f9f2367c249125984a7a158e9fcb0c1df110" exitCode=0
Nov 03 10:45:49.703439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:49.703021    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v" event={"ID":"0983ec40-cd53-44fe-bfbf-7a42f00e52f6","Type":"ContainerDied","Data":"179bded08410c50380cce5e19b42f9f2367c249125984a7a158e9fcb0c1df110"}
Nov 03 10:45:49.764726 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:49.764673    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:50.468764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:50.468733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:50.604944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:50.604912    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:50.764174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:50.764100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:50.898348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:50.898321    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:50.998201 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:50.998165    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-kube-api-access-zxtmw\" (UniqueName: \"kubernetes.io/projected/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-kube-api-access-zxtmw\") pod \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") "
Nov 03 10:45:51.012353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:50.998543    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-openshift-ca\") pod \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") "
Nov 03 10:45:51.012734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.012711    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-xks-ca\") pod \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\" (UID: \"0983ec40-cd53-44fe-bfbf-7a42f00e52f6\") "
Nov 03 10:45:51.014344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.007626    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-kube-api-access-zxtmw" pod "0983ec40-cd53-44fe-bfbf-7a42f00e52f6" (UID: "0983ec40-cd53-44fe-bfbf-7a42f00e52f6"). InnerVolumeSpecName "kube-api-access-zxtmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:51.014475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.008099    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-openshift-ca" pod "0983ec40-cd53-44fe-bfbf-7a42f00e52f6" (UID: "0983ec40-cd53-44fe-bfbf-7a42f00e52f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:51.018200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.018171    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-xks-ca" pod "0983ec40-cd53-44fe-bfbf-7a42f00e52f6" (UID: "0983ec40-cd53-44fe-bfbf-7a42f00e52f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:51.088752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.083785    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"]
Nov 03 10:45:51.088752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.086948    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0983ec40-cd53-44fe-bfbf-7a42f00e52f6" containerName="revoke-keys"
Nov 03 10:45:51.088752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.086973    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0983ec40-cd53-44fe-bfbf-7a42f00e52f6" containerName="revoke-keys"
Nov 03 10:45:51.088752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.087378    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0983ec40-cd53-44fe-bfbf-7a42f00e52f6" containerName="revoke-keys"
Nov 03 10:45:51.094167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.093401    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.105718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.105187    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"]
Nov 03 10:45:51.121121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.121084    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.121288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.121173    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.121288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.121271    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx5m2\" (UniqueName: \"kubernetes.io/projected/105d6801-0746-49de-a3c9-3a9252277f3f-kube-api-access-nx5m2\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.121414 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.121395    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxtmw\" (UniqueName: \"kubernetes.io/projected/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-kube-api-access-zxtmw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:51.121474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.121421    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:51.121474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.121435    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0983ec40-cd53-44fe-bfbf-7a42f00e52f6-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:51.222772 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.222739    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.222954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.222804    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.222954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.222882    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5m2\" (UniqueName: \"kubernetes.io/projected/105d6801-0746-49de-a3c9-3a9252277f3f-kube-api-access-nx5m2\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.223472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.223450    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.225213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.225189    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.231203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.231179    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5m2\" (UniqueName: \"kubernetes.io/projected/105d6801-0746-49de-a3c9-3a9252277f3f-kube-api-access-nx5m2\") pod \"maas-api-revoke-sub-b1f81728-m5pmh\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.420422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.420349    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:51.559363 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.559326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:51.705377 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:51.705349    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105d6801_0746_49de_a3c9_3a9252277f3f.slice/crio-4fad1067846ce59fb037610972d331e48e2125aeb60084ce884c983afd480969 WatchSource:0}: Error finding container 4fad1067846ce59fb037610972d331e48e2125aeb60084ce884c983afd480969: Status 404 returned error can't find the container with id 4fad1067846ce59fb037610972d331e48e2125aeb60084ce884c983afd480969
Nov 03 10:45:51.706194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.706162    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"]
Nov 03 10:45:51.740387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.740350    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh" event={"ID":"105d6801-0746-49de-a3c9-3a9252277f3f","Type":"ContainerStarted","Data":"4fad1067846ce59fb037610972d331e48e2125aeb60084ce884c983afd480969"}
Nov 03 10:45:51.745565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.745538    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v" event={"ID":"0983ec40-cd53-44fe-bfbf-7a42f00e52f6","Type":"ContainerDied","Data":"35370a032d2e09a5dea7bff5960d18407bd3f05ee762c70c6ee9815cbaee6f7c"}
Nov 03 10:45:51.745763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.745743    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35370a032d2e09a5dea7bff5960d18407bd3f05ee762c70c6ee9815cbaee6f7c"
Nov 03 10:45:51.745866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:51.745852    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"
Nov 03 10:45:52.495204 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:45:52.495160    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:45:52.677811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:52.677761    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:52.758304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:52.758215    5817 generic.go:356] "Generic (PLEG): container finished" podID="105d6801-0746-49de-a3c9-3a9252277f3f" containerID="605e01e9136a5716a4a57e0a2ea6b6c48a2b1931e425ffb6a67753025239e92a" exitCode=0
Nov 03 10:45:52.758485 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:52.758307    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh" event={"ID":"105d6801-0746-49de-a3c9-3a9252277f3f","Type":"ContainerDied","Data":"605e01e9136a5716a4a57e0a2ea6b6c48a2b1931e425ffb6a67753025239e92a"}
Nov 03 10:45:53.556098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.556065    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:53.775827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.774009    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:53.959095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.959071    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:53.973746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.973724    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-openshift-ca\") pod \"105d6801-0746-49de-a3c9-3a9252277f3f\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") "
Nov 03 10:45:53.973871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.973798    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-xks-ca\") pod \"105d6801-0746-49de-a3c9-3a9252277f3f\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") "
Nov 03 10:45:53.973914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.973889    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105d6801-0746-49de-a3c9-3a9252277f3f-kube-api-access-nx5m2\" (UniqueName: \"kubernetes.io/projected/105d6801-0746-49de-a3c9-3a9252277f3f-kube-api-access-nx5m2\") pod \"105d6801-0746-49de-a3c9-3a9252277f3f\" (UID: \"105d6801-0746-49de-a3c9-3a9252277f3f\") "
Nov 03 10:45:53.974207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.974180    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-openshift-ca" pod "105d6801-0746-49de-a3c9-3a9252277f3f" (UID: "105d6801-0746-49de-a3c9-3a9252277f3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:53.975243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.974694    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:53.977505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.977468    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-xks-ca" pod "105d6801-0746-49de-a3c9-3a9252277f3f" (UID: "105d6801-0746-49de-a3c9-3a9252277f3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:53.977833 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:53.977805    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105d6801-0746-49de-a3c9-3a9252277f3f-kube-api-access-nx5m2" pod "105d6801-0746-49de-a3c9-3a9252277f3f" (UID: "105d6801-0746-49de-a3c9-3a9252277f3f"). InnerVolumeSpecName "kube-api-access-nx5m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:54.076113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:54.076080    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105d6801-0746-49de-a3c9-3a9252277f3f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:54.076113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:54.076111    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx5m2\" (UniqueName: \"kubernetes.io/projected/105d6801-0746-49de-a3c9-3a9252277f3f-kube-api-access-nx5m2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:54.257552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:54.257445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:54.778862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:54.778834    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"
Nov 03 10:45:54.778862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:54.778856    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh" event={"ID":"105d6801-0746-49de-a3c9-3a9252277f3f","Type":"ContainerDied","Data":"4fad1067846ce59fb037610972d331e48e2125aeb60084ce884c983afd480969"}
Nov 03 10:45:54.779367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:54.778898    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fad1067846ce59fb037610972d331e48e2125aeb60084ce884c983afd480969"
Nov 03 10:45:54.963725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:54.963680    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:55.567696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:55.566208    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:55.853203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:55.853115    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:56.543522 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:56.543468    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:57.257067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.257034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:57.510399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.510317    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"]
Nov 03 10:45:57.511739 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.511640    5817 cpu_manager.go:414] "containerMap: removing container" podUID="105d6801-0746-49de-a3c9-3a9252277f3f" containerName="revoke-keys"
Nov 03 10:45:57.511937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.511922    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105d6801-0746-49de-a3c9-3a9252277f3f" containerName="revoke-keys"
Nov 03 10:45:57.512221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.512199    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="105d6801-0746-49de-a3c9-3a9252277f3f" containerName="revoke-keys"
Nov 03 10:45:57.518665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.516873    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.518665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.518087    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"]
Nov 03 10:45:57.523067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.522950    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:45:57.523067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.523021    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:57.636693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.636658    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm98g\" (UniqueName: \"kubernetes.io/projected/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-kube-api-access-wm98g\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.636895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.636797    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.636895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.636838    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.737827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.737790    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm98g\" (UniqueName: \"kubernetes.io/projected/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-kube-api-access-wm98g\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.738139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.738104    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.738271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.738166    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.738796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.738774    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.741617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.741595    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.746500 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.746478    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm98g\" (UniqueName: \"kubernetes.io/projected/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-kube-api-access-wm98g\") pod \"maas-api-revoke-sub-219808f0-29gmm\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.870058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.869982    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:45:57.999626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:57.999548    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:58.167783 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:45:58.167755    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f49dfd_7beb_4fdf_8c61_f61a06103f03.slice/crio-a94b3f0d6b0fcd224435f6446207454046dbddc1a02ce6557d6c0826530270b4 WatchSource:0}: Error finding container a94b3f0d6b0fcd224435f6446207454046dbddc1a02ce6557d6c0826530270b4: Status 404 returned error can't find the container with id a94b3f0d6b0fcd224435f6446207454046dbddc1a02ce6557d6c0826530270b4
Nov 03 10:45:58.170692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:58.170661    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"]
Nov 03 10:45:58.756356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:58.756317    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:58.831933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:58.831775    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm" event={"ID":"d8f49dfd-7beb-4fdf-8c61-f61a06103f03","Type":"ContainerStarted","Data":"0370429461ae455fce349468e235e561bdf2b5251b4bc30b6070aeafd19e0494"}
Nov 03 10:45:58.832259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:58.832222    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm" event={"ID":"d8f49dfd-7beb-4fdf-8c61-f61a06103f03","Type":"ContainerStarted","Data":"a94b3f0d6b0fcd224435f6446207454046dbddc1a02ce6557d6c0826530270b4"}
Nov 03 10:45:58.849830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:58.849748    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm" podStartSLOduration=1.8497347020000001 podStartE2EDuration="1.849734702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:58.848063148 +0000 UTC m=+989.053645201" watchObservedRunningTime="2026-11-03 10:45:58.849734702 +0000 UTC m=+989.055316786"
Nov 03 10:45:59.494601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:59.494510    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:45:59.562432 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:59.562373    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:45:59.724021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:59.721806    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:45:59.732553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:59.732529    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:45:59.845327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:59.845248    5817 generic.go:356] "Generic (PLEG): container finished" podID="d8f49dfd-7beb-4fdf-8c61-f61a06103f03" containerID="0370429461ae455fce349468e235e561bdf2b5251b4bc30b6070aeafd19e0494" exitCode=0
Nov 03 10:45:59.845327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:45:59.845304    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm" event={"ID":"d8f49dfd-7beb-4fdf-8c61-f61a06103f03","Type":"ContainerDied","Data":"0370429461ae455fce349468e235e561bdf2b5251b4bc30b6070aeafd19e0494"}
Nov 03 10:46:00.465712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:00.465679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:02.175598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.172929    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:46:02.268598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.268340    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.276598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.269692    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-xks-ca\") pod \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") "
Nov 03 10:46:02.276598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.269784    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-openshift-ca\") pod \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") "
Nov 03 10:46:02.276598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.269904    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-kube-api-access-wm98g\" (UniqueName: \"kubernetes.io/projected/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-kube-api-access-wm98g\") pod \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\" (UID: \"d8f49dfd-7beb-4fdf-8c61-f61a06103f03\") "
Nov 03 10:46:02.313605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.309883    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-openshift-ca" pod "d8f49dfd-7beb-4fdf-8c61-f61a06103f03" (UID: "d8f49dfd-7beb-4fdf-8c61-f61a06103f03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:46:02.358599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.356732    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-kube-api-access-wm98g" pod "d8f49dfd-7beb-4fdf-8c61-f61a06103f03" (UID: "d8f49dfd-7beb-4fdf-8c61-f61a06103f03"). InnerVolumeSpecName "kube-api-access-wm98g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:46:02.383600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.376092    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm98g\" (UniqueName: \"kubernetes.io/projected/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-kube-api-access-wm98g\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:46:02.383600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.376144    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:46:02.417600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.416686    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-xks-ca" pod "d8f49dfd-7beb-4fdf-8c61-f61a06103f03" (UID: "d8f49dfd-7beb-4fdf-8c61-f61a06103f03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:46:02.516594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.516262    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f49dfd-7beb-4fdf-8c61-f61a06103f03-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:46:02.553654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.552789    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.557597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.555562    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.577452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.563208    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.650597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.647991    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.650597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.648712    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.650597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.649619    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:46:02.674407 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.674297    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:46:02.838874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.836014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:02.915811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:02.861413    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.151448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.132190    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.151448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.146949    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.385261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.328491    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.395689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.389376    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm" event={"ID":"d8f49dfd-7beb-4fdf-8c61-f61a06103f03","Type":"ContainerDied","Data":"a94b3f0d6b0fcd224435f6446207454046dbddc1a02ce6557d6c0826530270b4"}
Nov 03 10:46:03.395689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.389442    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94b3f0d6b0fcd224435f6446207454046dbddc1a02ce6557d6c0826530270b4"
Nov 03 10:46:03.395689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.389523    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"
Nov 03 10:46:03.559725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.558672    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.560985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.560919    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:46:03.576618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.566587    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.577358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.577334    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:46:03.670598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.621076    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.670598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.621550    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:46:03.719232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.685655    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.719232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.688017    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:46:03.719232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.688838    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.719232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.689138    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.719232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.689428    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:46:03.719232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.689500    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:46:03.719232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.716905    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.719232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:03.717623    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:46:04.263669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.171806    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:04.263669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.172599    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:46:04.516272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.310536    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.540850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.387194    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.628598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.575678    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.656871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.656827    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:04.658621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.658589    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.725634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.694490    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.725986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.725949    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:04.735958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.735919    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.757763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:04.757719    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:46:05.159598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.159283    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded"
Nov 03 10:46:05.172224 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.172178    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.262073 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:46:05.262039    5817 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.089s"
Nov 03 10:46:05.262402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.262387    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:46:05.262533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.262523    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:46:05.510628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.474718    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:05.549616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.524906    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.608601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.607161    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.614677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.614637    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.677719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.677672    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.727212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.726878    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.754817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.735537    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.755320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.736594    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.790091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.790000    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:46:05.797597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.793917    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:46:05.864420 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.864358    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:46:05.864738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:05.864706    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:06.142658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.138986    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:46:06.217609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.172990    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:06.252034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.251984    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Nov 03 10:46:06.481533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.480633    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:46:06.527274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.525862    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:06.537125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.533274    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:46:06.537125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.534991    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:46:06.540764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.538028    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:46:06.540764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.538970    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:46:06.562948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.562924    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:46:06.573048 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.573028    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:46:06.624597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.616158    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:46:06.647754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.646798    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:46:06.654057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.654027    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:46:06.762473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.762379    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:06.944262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:06.944232    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:07.099535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:07.099440    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:07.866375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:07.866340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:08.237645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.237597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:08.871740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.867784    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"]
Nov 03 10:46:08.871740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.868883    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d8f49dfd-7beb-4fdf-8c61-f61a06103f03" containerName="revoke-keys"
Nov 03 10:46:08.871740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.868905    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f49dfd-7beb-4fdf-8c61-f61a06103f03" containerName="revoke-keys"
Nov 03 10:46:08.871740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.869081    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f49dfd-7beb-4fdf-8c61-f61a06103f03" containerName="revoke-keys"
Nov 03 10:46:08.925684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.890798    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"]
Nov 03 10:46:08.925684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.890930    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:08.925684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.918887    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-151744-openshift-default-dockercfg-jcljl\""
Nov 03 10:46:08.925684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:08.922596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:09.056827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.056717    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-certs\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.056827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.056796    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-data\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.057189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.057151    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-credential-socket\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.057353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.057297    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8v58\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-kube-api-access-h8v58\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.057589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.057521    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-token\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.057759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.057698    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-podinfo\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.058143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.058047    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18144-9897-4034-99ff-f5ddec642f0d-istiod-ca-cert\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.058143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.058106    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-envoy\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.058433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.058364    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-socket\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.119311 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.119275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:09.166946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.161146    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18144-9897-4034-99ff-f5ddec642f0d-istiod-ca-cert\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.166946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.164453    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18144-9897-4034-99ff-f5ddec642f0d-istiod-ca-cert\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.168409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.167243    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-envoy\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.168409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.167354    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-socket\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.168409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.167429    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-certs\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.168409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.167475    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-data\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.168409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.167518    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-credential-socket\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.168409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.167553    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8v58\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-kube-api-access-h8v58\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.168409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.167696    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-token\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.168409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.167760    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-podinfo\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.173511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.169947    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-credential-socket\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.173511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.170112    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-certs\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.173511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.170249    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-data\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.173511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.170353    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-socket\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.173511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.171141    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-podinfo\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.175793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.175755    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-envoy\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.182876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.182794    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-token\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.182876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.182861    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8v58\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-kube-api-access-h8v58\") pod \"e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") " pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.310597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.307851    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:09.327410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.327279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:09.670407 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:46:09.670371    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe18144_9897_4034_99ff_f5ddec642f0d.slice/crio-145a171c49176af6057f48cb055a8acd90961404b3a4e32052c2d4df5de7175a WatchSource:0}: Error finding container 145a171c49176af6057f48cb055a8acd90961404b3a4e32052c2d4df5de7175a: Status 404 returned error can't find the container with id 145a171c49176af6057f48cb055a8acd90961404b3a4e32052c2d4df5de7175a
Nov 03 10:46:09.670966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.670885    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"]
Nov 03 10:46:09.672963 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.672933    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:09.673058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.672996    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:09.673058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:09.673024    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:10.083946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:10.082345    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:10.544247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:10.544207    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" event={"ID":"ffe18144-9897-4034-99ff-f5ddec642f0d","Type":"ContainerStarted","Data":"b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e"}
Nov 03 10:46:10.544411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:10.544278    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" event={"ID":"ffe18144-9897-4034-99ff-f5ddec642f0d","Type":"ContainerStarted","Data":"145a171c49176af6057f48cb055a8acd90961404b3a4e32052c2d4df5de7175a"}
Nov 03 10:46:10.565315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:10.565270    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podStartSLOduration=2.565256125 podStartE2EDuration="2.565256125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:10.562976457 +0000 UTC m=+1000.768558509" watchObservedRunningTime="2026-11-03 10:46:10.565256125 +0000 UTC m=+1000.770838175"
Nov 03 10:46:10.752215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:10.750941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:10.901430 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:10.901341    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:11.309268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:11.309240    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:11.314782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:11.314758    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:11.557977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:11.557907    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:11.576025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:11.575908    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:11.578668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:11.578644    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:46:11.693166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:11.693129    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:12.048031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:12.047985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:12.203598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:12.193074    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:12.857388 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:12.857357    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:13.005781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:13.005744    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:13.863708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:13.863487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:14.208229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.208196    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:14.227944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.227913    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"]
Nov 03 10:46:14.245928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.245894    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:14.246643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.245786    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"]
Nov 03 10:46:14.318147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.308768    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-151744\""
Nov 03 10:46:14.318147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.308928    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-151744-dockercfg-h689g\""
Nov 03 10:46:14.353535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.353459    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:14.361596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.361539    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5cz\" (UniqueName: \"kubernetes.io/projected/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-kube-api-access-tt5cz\") pod \"payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l\" (UID: \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:14.365591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.361897    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l\" (UID: \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:14.471692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.467632    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l\" (UID: \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:14.471692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.467772    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cz\" (UniqueName: \"kubernetes.io/projected/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-kube-api-access-tt5cz\") pod \"payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l\" (UID: \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:14.471692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.469289    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l\" (UID: \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:14.508353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.503297    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5cz\" (UniqueName: \"kubernetes.io/projected/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-kube-api-access-tt5cz\") pod \"payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l\" (UID: \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:14.582510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.569980    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:14.868506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.868480    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"]
Nov 03 10:46:14.920602 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:46:14.920545    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7555ea1a_4b05_4c03_aed7_eefdb6ab5c80.slice/crio-7e92ec78351eca5741f4d9e702baeada5939c608f6982aec4737e25c941a999c WatchSource:0}: Error finding container 7e92ec78351eca5741f4d9e702baeada5939c608f6982aec4737e25c941a999c: Status 404 returned error can't find the container with id 7e92ec78351eca5741f4d9e702baeada5939c608f6982aec4737e25c941a999c
Nov 03 10:46:14.930542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:14.930508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:15.206518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.206474    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"]
Nov 03 10:46:15.212067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.212041    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.227212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.227183    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"]
Nov 03 10:46:15.280563 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.280535    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.280743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.280619    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.280743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.280697    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb29c7e-ad83-4646-b10f-39f54cba1460-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.280820 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.280794    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxn6p\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-kube-api-access-pxn6p\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.280853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.280822    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.280889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.280879    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.280950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.280937    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.280986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.280968    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.281057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.281036    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.382788 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.382759    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb29c7e-ad83-4646-b10f-39f54cba1460-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.382870    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxn6p\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-kube-api-access-pxn6p\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.382916    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.382959    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.383065    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.383110    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.383149    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.383244    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.383315    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.383625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.383564    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb29c7e-ad83-4646-b10f-39f54cba1460-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.384036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.384013    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.384263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.384139    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.384689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.384394    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.385005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.384961    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.387150 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.387126    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.387385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.387366    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.392089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.392063    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxn6p\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-kube-api-access-pxn6p\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.392390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.392367    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-8pk88\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.539199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.539126    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:15.647286 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.647245    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l" event={"ID":"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80","Type":"ContainerStarted","Data":"0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b"}
Nov 03 10:46:15.647452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.647315    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l" event={"ID":"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80","Type":"ContainerStarted","Data":"7e92ec78351eca5741f4d9e702baeada5939c608f6982aec4737e25c941a999c"}
Nov 03 10:46:15.647452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.647370    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:15.649719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.649690    5817 generic.go:356] "Generic (PLEG): container finished" podID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerID="80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0" exitCode=0
Nov 03 10:46:15.649849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.649767    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" event={"ID":"d0086c91-894b-4b31-8eb7-7c147e4512ba","Type":"ContainerDied","Data":"80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0"}
Nov 03 10:46:15.650244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.650222    5817 scope.go:122] "RemoveContainer" containerID="80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0"
Nov 03 10:46:15.652044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.652016    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.652151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.652092    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.652151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.652140    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.673454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.673394    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l" podStartSLOduration=1.6733819749999999 podStartE2EDuration="1.673381975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:15.668189064 +0000 UTC m=+1005.873771116" watchObservedRunningTime="2026-11-03 10:46:15.673381975 +0000 UTC m=+1005.878964026"
Nov 03 10:46:15.766672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.766639    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:15.867752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.866879    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"]
Nov 03 10:46:15.879890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.879381    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.879890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.879467    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.879890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:15.879507    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:16.666698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:16.666651    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" event={"ID":"d0086c91-894b-4b31-8eb7-7c147e4512ba","Type":"ContainerStarted","Data":"30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361"}
Nov 03 10:46:16.669373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:16.669342    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" event={"ID":"efb29c7e-ad83-4646-b10f-39f54cba1460","Type":"ContainerStarted","Data":"742de57737f35ef1919d46dea7ebc471d72d5ed981decfba1bec87d460df671f"}
Nov 03 10:46:16.669509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:16.669409    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" event={"ID":"efb29c7e-ad83-4646-b10f-39f54cba1460","Type":"ContainerStarted","Data":"75d23bd68f66a6baef9aebee8f43611b415e0cb63dd903bf091d6378650bb604"}
Nov 03 10:46:16.681792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:16.675124    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:16.817598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:16.814651    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" podStartSLOduration=1.8146307 podStartE2EDuration="1.8146307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:16.716249202 +0000 UTC m=+1006.921831255" watchObservedRunningTime="2026-11-03 10:46:16.8146307 +0000 UTC m=+1007.020212752"
Nov 03 10:46:17.435873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:17.435740    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:46:17.438341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:17.438298    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 10:46:17.438592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:17.438536    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 10:46:17.461704 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:17.461664    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:17.551597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:17.545200    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:17.551597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:17.545337    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 10:46:17.551597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:17.545379    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 10:46:17.846647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:17.846247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:18.141649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:18.141237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:18.268375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:18.267902    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:46:18.299601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:18.295049    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:18.425026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:18.424940    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 10:46:18.425319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:18.425277    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 10:46:18.546689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:18.546657    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 10:46:18.547259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:18.547229    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 10:46:19.086311 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:19.085993    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:19.434648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:19.425085    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 10:46:19.434648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:19.425145    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 10:46:19.546607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:19.545939    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 10:46:19.546607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:19.545999    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 10:46:20.038598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:20.038507    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:20.170324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:20.170291    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:20.423655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:20.422969    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 10:46:20.423655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:20.423027    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 10:46:20.542596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:20.541992    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 10:46:20.542596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:20.542049    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 10:46:21.309603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:21.301538    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:21.425595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:21.424919    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 10:46:21.425595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:21.424980    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 10:46:21.543799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:21.539956    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 10:46:21.543799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:21.540058    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 10:46:22.190565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:22.190530    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:22.413406 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:46:22.413371    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:46:22.426836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:22.426811    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:46:22.427912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:22.427893    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:46:22.544007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:22.543919    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:22.544395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:22.544375    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:22.545109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:22.545092    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:46:22.977241 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:22.977183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:23.653528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:23.653492    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:23.786065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:23.786033    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:23.865660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:23.865630    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:46:24.355430 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:24.355389    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:25.898595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:25.898414    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:26.082641 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:26.079397    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:26.349596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:26.344423    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:26.703803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:26.703775    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:46:27.265653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.247912    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:27.440454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.437155    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"]
Nov 03 10:46:27.461597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.457016    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"]
Nov 03 10:46:27.461976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.461944    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:27.486989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.475730    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:27.497830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.494170    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a917b1\""
Nov 03 10:46:27.497830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.495507    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44ct\" (UniqueName: \"kubernetes.io/projected/08f290f8-ff27-49df-bc59-afefbe16608d-kube-api-access-d44ct\") pod \"payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg\" (UID: \"08f290f8-ff27-49df-bc59-afefbe16608d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:27.497830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.495727    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f290f8-ff27-49df-bc59-afefbe16608d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg\" (UID: \"08f290f8-ff27-49df-bc59-afefbe16608d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:27.497830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.495826    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndt5j\" (UniqueName: \"kubernetes.io/projected/e0ae0509-f41d-4eb4-aa40-2934662640c7-kube-api-access-ndt5j\") pod \"payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5\" (UID: \"e0ae0509-f41d-4eb4-aa40-2934662640c7\") " pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:27.497830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.495855    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ae0509-f41d-4eb4-aa40-2934662640c7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5\" (UID: \"e0ae0509-f41d-4eb4-aa40-2934662640c7\") " pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:27.558172 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.558082    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a917b1-dockercfg-qd7dr\""
Nov 03 10:46:27.565039 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.562768    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:46:27.603267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.602707    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"]
Nov 03 10:46:27.614064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.614013    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:27.623398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.623360    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndt5j\" (UniqueName: \"kubernetes.io/projected/e0ae0509-f41d-4eb4-aa40-2934662640c7-kube-api-access-ndt5j\") pod \"payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5\" (UID: \"e0ae0509-f41d-4eb4-aa40-2934662640c7\") " pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:27.623622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.623601    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ae0509-f41d-4eb4-aa40-2934662640c7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5\" (UID: \"e0ae0509-f41d-4eb4-aa40-2934662640c7\") " pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:27.623815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.623789    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44ct\" (UniqueName: \"kubernetes.io/projected/08f290f8-ff27-49df-bc59-afefbe16608d-kube-api-access-d44ct\") pod \"payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg\" (UID: \"08f290f8-ff27-49df-bc59-afefbe16608d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:27.624216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.624195    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f290f8-ff27-49df-bc59-afefbe16608d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg\" (UID: \"08f290f8-ff27-49df-bc59-afefbe16608d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:27.626100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.626078    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ae0509-f41d-4eb4-aa40-2934662640c7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5\" (UID: \"e0ae0509-f41d-4eb4-aa40-2934662640c7\") " pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:27.626310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.626281    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f290f8-ff27-49df-bc59-afefbe16608d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg\" (UID: \"08f290f8-ff27-49df-bc59-afefbe16608d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:27.659597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.659452    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"]
Nov 03 10:46:27.660683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.660663    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44ct\" (UniqueName: \"kubernetes.io/projected/08f290f8-ff27-49df-bc59-afefbe16608d-kube-api-access-d44ct\") pod \"payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg\" (UID: \"08f290f8-ff27-49df-bc59-afefbe16608d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:27.661360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.661339    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndt5j\" (UniqueName: \"kubernetes.io/projected/e0ae0509-f41d-4eb4-aa40-2934662640c7-kube-api-access-ndt5j\") pod \"payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5\" (UID: \"e0ae0509-f41d-4eb4-aa40-2934662640c7\") " pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:27.666188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.666166    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:27.772665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.772629    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:27.931494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:27.931461    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:28.069398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:28.069372    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"]
Nov 03 10:46:28.071208 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:46:28.071180    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f290f8_ff27_49df_bc59_afefbe16608d.slice/crio-a6277c85e776bc38baed540b12bc3e46afc074501a194da0863190690d371f39 WatchSource:0}: Error finding container a6277c85e776bc38baed540b12bc3e46afc074501a194da0863190690d371f39: Status 404 returned error can't find the container with id a6277c85e776bc38baed540b12bc3e46afc074501a194da0863190690d371f39
Nov 03 10:46:28.150011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:28.149980    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg" event={"ID":"08f290f8-ff27-49df-bc59-afefbe16608d","Type":"ContainerStarted","Data":"a6277c85e776bc38baed540b12bc3e46afc074501a194da0863190690d371f39"}
Nov 03 10:46:28.220167 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:46:28.220136    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ae0509_f41d_4eb4_aa40_2934662640c7.slice/crio-0704e758918ff60f84b4d556134628e2b3b273de685d0f376f368ccd27237d06 WatchSource:0}: Error finding container 0704e758918ff60f84b4d556134628e2b3b273de685d0f376f368ccd27237d06: Status 404 returned error can't find the container with id 0704e758918ff60f84b4d556134628e2b3b273de685d0f376f368ccd27237d06
Nov 03 10:46:28.222412 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:28.222379    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"]
Nov 03 10:46:28.376033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:28.375921    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:28.975486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:28.975100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:29.159288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:29.159247    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5" event={"ID":"e0ae0509-f41d-4eb4-aa40-2934662640c7","Type":"ContainerStarted","Data":"7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d"}
Nov 03 10:46:29.159488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:29.159312    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:29.159488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:29.159339    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5" event={"ID":"e0ae0509-f41d-4eb4-aa40-2934662640c7","Type":"ContainerStarted","Data":"0704e758918ff60f84b4d556134628e2b3b273de685d0f376f368ccd27237d06"}
Nov 03 10:46:29.161305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:29.161270    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg" event={"ID":"08f290f8-ff27-49df-bc59-afefbe16608d","Type":"ContainerStarted","Data":"628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8"}
Nov 03 10:46:29.161405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:29.161319    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:29.179723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:29.179674    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5" podStartSLOduration=2.17965866 podStartE2EDuration="2.17965866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:29.174628525 +0000 UTC m=+1019.380210588" watchObservedRunningTime="2026-11-03 10:46:29.17965866 +0000 UTC m=+1019.385240711"
Nov 03 10:46:29.914895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:29.914863    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:30.267816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:30.267786    5817 scope.go:122] "RemoveContainer" containerID="4d42415af7bfa58a3d79c213454d7ad6d1bdf8d71a46ec9d09bffbf376a88642"
Nov 03 10:46:30.278694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:30.278673    5817 scope.go:122] "RemoveContainer" containerID="6817673764fdd1f495ff48df640361025d0d36a452e128122251b1feb8bb94f2"
Nov 03 10:46:30.549547 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:30.549458    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:31.156209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:31.156172    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:31.757996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:31.757962    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:32.379012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:32.378977    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:32.549029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:32.548990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:32.994951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:32.994913    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:33.149628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:33.145073    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:33.821040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:33.821003    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:34.281190 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.281147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:34.326916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.326862    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg" podStartSLOduration=7.326847983 podStartE2EDuration="7.326847983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:29.190310409 +0000 UTC m=+1019.395892463" watchObservedRunningTime="2026-11-03 10:46:34.326847983 +0000 UTC m=+1024.532430034"
Nov 03 10:46:34.330380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.330357    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"]
Nov 03 10:46:34.336424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.336406    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.349825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.349802    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 10:46:34.349947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.349862    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 10:46:34.349947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.349939    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fbzlj\""
Nov 03 10:46:34.350055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.349982    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 10:46:34.352743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.352716    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"]
Nov 03 10:46:34.494627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.486528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:34.598923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.598823    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c363a74-0ed9-47ba-9902-0345252cda60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.599101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.598993    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9khs\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-kube-api-access-v9khs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.599165 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.599107    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.599220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.599179    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.599325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.599307    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.599438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.599422    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.599498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.599469    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.701540 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.701497    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.701756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.701659    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.701828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.701802    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.701881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.701861    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.701933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.701910    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c363a74-0ed9-47ba-9902-0345252cda60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.702162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.702141    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9khs\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-kube-api-access-v9khs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.702253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.702241    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.702679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.702659    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.705024 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.704999    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c363a74-0ed9-47ba-9902-0345252cda60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.705538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.705517    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.705664 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.705517    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.706971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.706924    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.710672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.710585    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9khs\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-kube-api-access-v9khs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.711842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.711821    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-94kvg\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:34.950555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:34.950266    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:35.254645 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:46:35.254613    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c363a74_0ed9_47ba_9902_0345252cda60.slice/crio-17efca0260bc6c53f2ebb1c914983400b1eb54ca520660839595b23d00274256 WatchSource:0}: Error finding container 17efca0260bc6c53f2ebb1c914983400b1eb54ca520660839595b23d00274256: Status 404 returned error can't find the container with id 17efca0260bc6c53f2ebb1c914983400b1eb54ca520660839595b23d00274256
Nov 03 10:46:35.257079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:35.257063    5817 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:46:35.257992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:35.257967    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"]
Nov 03 10:46:35.304594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:35.300421    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:35.438827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:35.438795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:36.205934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:36.205895    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:36.269696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:36.269660    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg" event={"ID":"4c363a74-0ed9-47ba-9902-0345252cda60","Type":"ContainerStarted","Data":"17efca0260bc6c53f2ebb1c914983400b1eb54ca520660839595b23d00274256"}
Nov 03 10:46:37.088997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:37.088957    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:37.344401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:37.344320    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:37.479741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:37.479696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:38.122520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:38.122493    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:38.347177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:38.347132    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:38.435638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:38.435608    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:38.435755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:38.435673    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:38.611290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:38.611260    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:38.786121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:38.786043    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:39.303828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:39.303787    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg" event={"ID":"4c363a74-0ed9-47ba-9902-0345252cda60","Type":"ContainerStarted","Data":"f3da7c43780d558de06b890a4d4a0775da9c4f58aef1f7a1dc48eb07c4f5cd46"}
Nov 03 10:46:39.304071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:39.303940    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:39.305973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:39.305945    5817 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-94kvg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:46:39.306231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:39.306204    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg" podUID="4c363a74-0ed9-47ba-9902-0345252cda60" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:46:39.336068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:39.336009    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg" podStartSLOduration=2.158852823 podStartE2EDuration="5.335983473s" totalImagesPullingTime="3.17713065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:39.333599057 +0000 UTC m=+1029.539181106" watchObservedRunningTime="2026-11-03 10:46:39.335983473 +0000 UTC m=+1029.541565525"
Nov 03 10:46:39.579302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:39.579220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:39.768597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:39.754736    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:40.174587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:40.174534    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:46:40.188793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:40.188761    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:46:40.318884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:40.318856    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:46:40.662400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:40.661737    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:41.378077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:41.378042    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:41.514260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:41.514225    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:42.093193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:42.093158    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:42.699852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:42.699791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:43.279742 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:43.279705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:43.885580 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:43.885529    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:44.551019 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:44.550982    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:44.693210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:44.693178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:45.350348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.350312    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"]
Nov 03 10:46:45.355849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.355826    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.369355 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.369318    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"]
Nov 03 10:46:45.439422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.439381    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a691d134-0c27-4517-ba67-d76ab40cd5a6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.439609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.439522    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.439721 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.439697    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.439775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.439761    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2nw\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-kube-api-access-hh2nw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.439854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.439838    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.439904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.439879    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.439984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.439969    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.440030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.440013    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.440073 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.440034    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.520918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.520275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:45.546463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546416    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.546702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546547    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.546702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546640    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2nw\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-kube-api-access-hh2nw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.546702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546695    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.546880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546733    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.546880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546826    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.546880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546869    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.547030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546888    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.547030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546957    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a691d134-0c27-4517-ba67-d76ab40cd5a6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.547030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546977    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.547030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.546993    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.547632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.547604    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.547915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.547895    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.548149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.548126    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a691d134-0c27-4517-ba67-d76ab40cd5a6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.550316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.550297    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.550409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.550373    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.557821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.557797    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.561172 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.561149    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2nw\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-kube-api-access-hh2nw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.676724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.676639    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:45.982813 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:46:45.982783    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda691d134_0c27_4517_ba67_d76ab40cd5a6.slice/crio-9a45c650eb8d3fb8108659e90af8b1afde072a9d34cd33cb93dfa11f2606c8d4 WatchSource:0}: Error finding container 9a45c650eb8d3fb8108659e90af8b1afde072a9d34cd33cb93dfa11f2606c8d4: Status 404 returned error can't find the container with id 9a45c650eb8d3fb8108659e90af8b1afde072a9d34cd33cb93dfa11f2606c8d4
Nov 03 10:46:45.983826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.983803    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"]
Nov 03 10:46:45.985804 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.985766    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:45.985886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.985848    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:45.985927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:45.985892    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:46.330359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:46.330281    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:46.378956 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:46.378923    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" event={"ID":"a691d134-0c27-4517-ba67-d76ab40cd5a6","Type":"ContainerStarted","Data":"a328e82241cacc104649f4d44d20ea61014e90af74b49667f0db015b827140cf"}
Nov 03 10:46:46.379158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:46.378990    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" event={"ID":"a691d134-0c27-4517-ba67-d76ab40cd5a6","Type":"ContainerStarted","Data":"9a45c650eb8d3fb8108659e90af8b1afde072a9d34cd33cb93dfa11f2606c8d4"}
Nov 03 10:46:46.398696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:46.398639    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podStartSLOduration=1.398622576 podStartE2EDuration="1.398622576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:46.396634118 +0000 UTC m=+1036.602216166" watchObservedRunningTime="2026-11-03 10:46:46.398622576 +0000 UTC m=+1036.604204628"
Nov 03 10:46:46.468707 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:46.468672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:46.677487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:46.677410    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:46.781660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:46.781621    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:47.368597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:47.365643    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:47.679352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:47.678636    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:46:47.679352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:47.678685    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 10:46:48.383038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:48.383006    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:48.682594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:48.682008    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:46:48.682594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:48.682090    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 10:46:49.449947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:49.449915    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:49.650605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:49.650185    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:49.677145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:49.677109    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:46:49.677357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:49.677164    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 10:46:50.677871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:50.677824    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:46:50.678486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:50.677893    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 10:46:50.824597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:50.809939    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:51.677429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:51.677395    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:46:51.677656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:51.677448    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 10:46:51.949702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:51.945279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:52.466942 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:46:52.466889    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:46:52.677986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:52.677945    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:46:52.678165 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:52.678011    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 10:46:53.114503 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:53.114470    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:53.695872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:53.695843    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:53.697632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:53.697613    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:53.706019 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:53.706002    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:46:54.436549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:54.436462    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:55.507616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:55.503129    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:55.639707 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:55.639678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:56.175881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:56.175847    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:56.937380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:56.937277    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:57.615756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:57.615708    5817 patch_prober.go:29] interesting pod/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:46:57.615962 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:57.615874    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:57.722595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:57.715216    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:58.541123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.541085    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"]
Nov 03 10:46:58.553718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.553687    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.558076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.558055    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:46:58.558920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.558901    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:46:58.572828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.572523    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:58.579105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.579019    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"]
Nov 03 10:46:58.675062 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.675015    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.675256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.675135    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42gp\" (UniqueName: \"kubernetes.io/projected/96aa703c-e838-48c3-8dc7-185240898d94-kube-api-access-k42gp\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.675256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.675221    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.777204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.777158    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.777402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.777255    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42gp\" (UniqueName: \"kubernetes.io/projected/96aa703c-e838-48c3-8dc7-185240898d94-kube-api-access-k42gp\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.777402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.777325    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.778004 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.777981    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.779738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.779719    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.785158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.785135    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42gp\" (UniqueName: \"kubernetes.io/projected/96aa703c-e838-48c3-8dc7-185240898d94-kube-api-access-k42gp\") pod \"maas-api-revoke-keys-e2e-api-a-141053-w8bng\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:58.901456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:58.901388    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:46:59.190367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.190339    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"]
Nov 03 10:46:59.201810 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:46:59.201781    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96aa703c_e838_48c3_8dc7_185240898d94.slice/crio-74a008f469ff2d1b66028a3ce8edacaea96af45e20b9b1ea152a7204904930d0 WatchSource:0}: Error finding container 74a008f469ff2d1b66028a3ce8edacaea96af45e20b9b1ea152a7204904930d0: Status 404 returned error can't find the container with id 74a008f469ff2d1b66028a3ce8edacaea96af45e20b9b1ea152a7204904930d0
Nov 03 10:46:59.424014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.423965    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:46:59.687323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.687282    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng" event={"ID":"96aa703c-e838-48c3-8dc7-185240898d94","Type":"ContainerStarted","Data":"f60209036d2b2770ac0e04affb68e3489b021f35192fcbbd2517bd0997911875"}
Nov 03 10:46:59.687845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.687349    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng" event={"ID":"96aa703c-e838-48c3-8dc7-185240898d94","Type":"ContainerStarted","Data":"74a008f469ff2d1b66028a3ce8edacaea96af45e20b9b1ea152a7204904930d0"}
Nov 03 10:46:59.689407 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.689380    5817 generic.go:356] "Generic (PLEG): container finished" podID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerID="b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e" exitCode=0
Nov 03 10:46:59.689534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.689456    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" event={"ID":"ffe18144-9897-4034-99ff-f5ddec642f0d","Type":"ContainerDied","Data":"b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e"}
Nov 03 10:46:59.689969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.689952    5817 scope.go:122] "RemoveContainer" containerID="b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e"
Nov 03 10:46:59.691815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.691788    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:59.691938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.691855    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:59.691938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.691884    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:59.706395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:46:59.705525    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng" podStartSLOduration=1.7055090929999999 podStartE2EDuration="1.705509093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:59.702289746 +0000 UTC m=+1049.907871797" watchObservedRunningTime="2026-11-03 10:46:59.705509093 +0000 UTC m=+1049.911091147"
Nov 03 10:47:00.194858 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:00.194736    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:00.719119 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:00.719089    5817 generic.go:356] "Generic (PLEG): container finished" podID="96aa703c-e838-48c3-8dc7-185240898d94" containerID="f60209036d2b2770ac0e04affb68e3489b021f35192fcbbd2517bd0997911875" exitCode=0
Nov 03 10:47:00.719789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:00.719764    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng" event={"ID":"96aa703c-e838-48c3-8dc7-185240898d94","Type":"ContainerDied","Data":"f60209036d2b2770ac0e04affb68e3489b021f35192fcbbd2517bd0997911875"}
Nov 03 10:47:00.724596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:00.723908    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" event={"ID":"ffe18144-9897-4034-99ff-f5ddec642f0d","Type":"ContainerStarted","Data":"5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515"}
Nov 03 10:47:01.056840 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:01.055283    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:01.310876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:01.310797    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:47:01.311250 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:01.311228    5817 patch_prober.go:29] interesting pod/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 10:47:01.311411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:01.311389    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 10:47:01.940198 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:01.937696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:02.067019 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.066991    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:47:02.169301 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.169272    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-openshift-ca\") pod \"96aa703c-e838-48c3-8dc7-185240898d94\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") "
Nov 03 10:47:02.169490 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.169441    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-xks-ca\") pod \"96aa703c-e838-48c3-8dc7-185240898d94\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") "
Nov 03 10:47:02.169615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.169499    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96aa703c-e838-48c3-8dc7-185240898d94-kube-api-access-k42gp\" (UniqueName: \"kubernetes.io/projected/96aa703c-e838-48c3-8dc7-185240898d94-kube-api-access-k42gp\") pod \"96aa703c-e838-48c3-8dc7-185240898d94\" (UID: \"96aa703c-e838-48c3-8dc7-185240898d94\") "
Nov 03 10:47:02.169683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.169665    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-openshift-ca" pod "96aa703c-e838-48c3-8dc7-185240898d94" (UID: "96aa703c-e838-48c3-8dc7-185240898d94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:02.170197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.170095    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:02.171975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.171673    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-xks-ca" pod "96aa703c-e838-48c3-8dc7-185240898d94" (UID: "96aa703c-e838-48c3-8dc7-185240898d94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:47:02.171975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.171929    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96aa703c-e838-48c3-8dc7-185240898d94-kube-api-access-k42gp" pod "96aa703c-e838-48c3-8dc7-185240898d94" (UID: "96aa703c-e838-48c3-8dc7-185240898d94"). InnerVolumeSpecName "kube-api-access-k42gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:02.271340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.271254    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96aa703c-e838-48c3-8dc7-185240898d94-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:02.271340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.271287    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42gp\" (UniqueName: \"kubernetes.io/projected/96aa703c-e838-48c3-8dc7-185240898d94-kube-api-access-k42gp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:02.313089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.313047    5817 patch_prober.go:29] interesting pod/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 10:47:02.313280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.313118    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 10:47:02.565422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.565338    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:02.759391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.759362    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"
Nov 03 10:47:02.759391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.759373    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng" event={"ID":"96aa703c-e838-48c3-8dc7-185240898d94","Type":"ContainerDied","Data":"74a008f469ff2d1b66028a3ce8edacaea96af45e20b9b1ea152a7204904930d0"}
Nov 03 10:47:02.759627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.759414    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a008f469ff2d1b66028a3ce8edacaea96af45e20b9b1ea152a7204904930d0"
Nov 03 10:47:02.919694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:02.914010    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:03.197597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.186795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:03.313335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.313301    5817 patch_prober.go:29] interesting pod/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 10:47:03.313536 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.313360    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 10:47:03.526425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.525966    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"]
Nov 03 10:47:03.526797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.526776    5817 cpu_manager.go:414] "containerMap: removing container" podUID="96aa703c-e838-48c3-8dc7-185240898d94" containerName="revoke-keys"
Nov 03 10:47:03.526907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.526801    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96aa703c-e838-48c3-8dc7-185240898d94" containerName="revoke-keys"
Nov 03 10:47:03.526977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.526958    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="96aa703c-e838-48c3-8dc7-185240898d94" containerName="revoke-keys"
Nov 03 10:47:03.546109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.545917    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"]
Nov 03 10:47:03.546109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.546013    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.549925 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.549901    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-642fe5-openshift-default-dockercfg-hcgjg\""
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.595557    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-socket\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.595655    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-token\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.595722    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-envoy\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.595806    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9nf\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-kube-api-access-tx9nf\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.595844    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-data\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.595897    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-certs\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.596034    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39912f-e2d5-4961-a3f4-42d08009885a-istiod-ca-cert\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.596077    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-podinfo\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.599597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.596158    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-credential-socket\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.629412 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.629383    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.697701    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-socket\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.697769    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-token\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.697838    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-envoy\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.697919    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9nf\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-kube-api-access-tx9nf\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.697958    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-data\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.698010    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-certs\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.698137    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39912f-e2d5-4961-a3f4-42d08009885a-istiod-ca-cert\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.698176    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-podinfo\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.698256    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-credential-socket\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.698716    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-credential-socket\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.705639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.698971    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-socket\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.710665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.707031    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-certs\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.710665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.707525    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-data\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.710665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.708235    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39912f-e2d5-4961-a3f4-42d08009885a-istiod-ca-cert\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.710665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.710326    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-token\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.719946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.719918    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-podinfo\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.722982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.722950    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-envoy\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.727936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.727031    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9nf\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-kube-api-access-tx9nf\") pod \"e2e-api-b-642fe5-openshift-default-76fb996475-967bc\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") " pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.770048 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.770024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:03.871803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.870250    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:03.930600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:03.922519    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:04.317665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.310991    5817 patch_prober.go:29] interesting pod/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 10:47:04.317665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.311053    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 10:47:04.369999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.369864    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:04.369999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.369949    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:04.369999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.369967    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"]
Nov 03 10:47:04.369999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.369993    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:04.779932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.779895    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" event={"ID":"0d39912f-e2d5-4961-a3f4-42d08009885a","Type":"ContainerStarted","Data":"2ae754ea1635d766e917c64d1f1daf6188d5535d8d242441ab9348e41627cd34"}
Nov 03 10:47:04.780147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.779946    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" event={"ID":"0d39912f-e2d5-4961-a3f4-42d08009885a","Type":"ContainerStarted","Data":"527fbf4f155a2c4b211142f39a2e07f5b83ffc9559b98a7de04ed33fb5f34884"}
Nov 03 10:47:04.811598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.811397    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" podStartSLOduration=1.811373976 podStartE2EDuration="1.811373976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:04.800190837 +0000 UTC m=+1055.005772889" watchObservedRunningTime="2026-11-03 10:47:04.811373976 +0000 UTC m=+1055.016956067"
Nov 03 10:47:04.837282 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.835376    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:04.874619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:04.871251    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:05.312039 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:05.312006    5817 patch_prober.go:29] interesting pod/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 10:47:05.312246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:05.312064    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 10:47:05.819359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:05.819324    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:05.878411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:05.878384    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:06.313711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:06.313682    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:47:06.314212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:06.314192    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:47:06.314969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:06.314944    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:47:06.709682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:06.709653    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:06.836357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:06.836327    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:06.837690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:06.837657    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:07.578921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:07.578889    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:08.027185 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:08.027152    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"]
Nov 03 10:47:08.033164 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:08.033138    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b3eabe-zbznk"]
Nov 03 10:47:08.299963 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:08.299824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:08.647947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:08.647853    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"]
Nov 03 10:47:08.661498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:08.661472    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"]
Nov 03 10:47:08.770753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:08.770717    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:08.915652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:08.915361    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:09.114444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:09.114407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:09.879439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:09.879386    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:09.915242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:09.915192    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe18c3b-7ae3-4b84-8647-a44c517fbccb" path="/var/lib/kubelet/pods/fbe18c3b-7ae3-4b84-8647-a44c517fbccb/volumes"
Nov 03 10:47:10.607332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:10.607297    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:11.217412 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:11.217382    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:11.849382 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:11.849348    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:12.445894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:12.445849    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:13.170871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.170819    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:13.482299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.482240    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:13.649018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.648980    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2" podUID="29c3c4c4-6087-4653-9e86-8ce491859398" containerName="payload-processing" containerID="cri-o://3a12cbcf8dc27588729020f07c178d9e60b2c9d062b394280af23b40468c7757" gracePeriod=25
Nov 03 10:47:13.662492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.662460    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk" podUID="b5b651e1-3a26-4b8b-92b5-7f76a2aa1853" containerName="payload-pre-processing" containerID="cri-o://4503f8f2bd25ec34382c8cbbdde5ed0205fb54dcb1d64bb1ea714caab018e11c" gracePeriod=25
Nov 03 10:47:13.870670 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.869060    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:13.971338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.970821    5817 generic.go:356] "Generic (PLEG): container finished" podID="29c3c4c4-6087-4653-9e86-8ce491859398" containerID="3a12cbcf8dc27588729020f07c178d9e60b2c9d062b394280af23b40468c7757" exitCode=0
Nov 03 10:47:13.974071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.974037    5817 generic.go:356] "Generic (PLEG): container finished" podID="b5b651e1-3a26-4b8b-92b5-7f76a2aa1853" containerID="4503f8f2bd25ec34382c8cbbdde5ed0205fb54dcb1d64bb1ea714caab018e11c" exitCode=0
Nov 03 10:47:13.976142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.976109    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2" event={"ID":"29c3c4c4-6087-4653-9e86-8ce491859398","Type":"ContainerDied","Data":"3a12cbcf8dc27588729020f07c178d9e60b2c9d062b394280af23b40468c7757"}
Nov 03 10:47:13.976266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:13.976172    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk" event={"ID":"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853","Type":"ContainerDied","Data":"4503f8f2bd25ec34382c8cbbdde5ed0205fb54dcb1d64bb1ea714caab018e11c"}
Nov 03 10:47:14.023558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.016360    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:47:14.056852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.056830    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:47:14.141745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.141716    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c3c4c4-6087-4653-9e86-8ce491859398-kube-api-access-gxdxh\" (UniqueName: \"kubernetes.io/projected/29c3c4c4-6087-4653-9e86-8ce491859398-kube-api-access-gxdxh\") pod \"29c3c4c4-6087-4653-9e86-8ce491859398\" (UID: \"29c3c4c4-6087-4653-9e86-8ce491859398\") "
Nov 03 10:47:14.141903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.141861    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-kube-api-access-hp5bh\" (UniqueName: \"kubernetes.io/projected/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-kube-api-access-hp5bh\") pod \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\" (UID: \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\") "
Nov 03 10:47:14.141995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.141980    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c3c4c4-6087-4653-9e86-8ce491859398-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c3c4c4-6087-4653-9e86-8ce491859398-plugins-config-volume\") pod \"29c3c4c4-6087-4653-9e86-8ce491859398\" (UID: \"29c3c4c4-6087-4653-9e86-8ce491859398\") "
Nov 03 10:47:14.142098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.142076    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-plugins-config-volume\") pod \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\" (UID: \"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853\") "
Nov 03 10:47:14.142386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.142359    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c3c4c4-6087-4653-9e86-8ce491859398-plugins-config-volume" pod "29c3c4c4-6087-4653-9e86-8ce491859398" (UID: "29c3c4c4-6087-4653-9e86-8ce491859398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:14.142478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.142432    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-plugins-config-volume" pod "b5b651e1-3a26-4b8b-92b5-7f76a2aa1853" (UID: "b5b651e1-3a26-4b8b-92b5-7f76a2aa1853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:14.142793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.142774    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c3c4c4-6087-4653-9e86-8ce491859398-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:14.142872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.142793    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:14.144048 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.144026    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c3c4c4-6087-4653-9e86-8ce491859398-kube-api-access-gxdxh" pod "29c3c4c4-6087-4653-9e86-8ce491859398" (UID: "29c3c4c4-6087-4653-9e86-8ce491859398"). InnerVolumeSpecName "kube-api-access-gxdxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:14.144264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.144246    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-kube-api-access-hp5bh" pod "b5b651e1-3a26-4b8b-92b5-7f76a2aa1853" (UID: "b5b651e1-3a26-4b8b-92b5-7f76a2aa1853"). InnerVolumeSpecName "kube-api-access-hp5bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:14.214129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.214103    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"]
Nov 03 10:47:14.214989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.214968    5817 cpu_manager.go:414] "containerMap: removing container" podUID="29c3c4c4-6087-4653-9e86-8ce491859398" containerName="payload-processing"
Nov 03 10:47:14.214989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.214989    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c3c4c4-6087-4653-9e86-8ce491859398" containerName="payload-processing"
Nov 03 10:47:14.215151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.215031    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b5b651e1-3a26-4b8b-92b5-7f76a2aa1853" containerName="payload-pre-processing"
Nov 03 10:47:14.215151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.215039    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b651e1-3a26-4b8b-92b5-7f76a2aa1853" containerName="payload-pre-processing"
Nov 03 10:47:14.215255 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.215208    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b651e1-3a26-4b8b-92b5-7f76a2aa1853" containerName="payload-pre-processing"
Nov 03 10:47:14.215255 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.215238    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c3c4c4-6087-4653-9e86-8ce491859398" containerName="payload-processing"
Nov 03 10:47:14.219385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.219315    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.226378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.226357    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"]
Nov 03 10:47:14.228942 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.228921    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:47:14.229059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.228973    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:47:14.244034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.243974    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5bh\" (UniqueName: \"kubernetes.io/projected/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853-kube-api-access-hp5bh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:14.244034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.243998    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxdxh\" (UniqueName: \"kubernetes.io/projected/29c3c4c4-6087-4653-9e86-8ce491859398-kube-api-access-gxdxh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:14.345147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.345110    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcmvl\" (UniqueName: \"kubernetes.io/projected/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-kube-api-access-fcmvl\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.345315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.345185    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.345315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.345281    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.446132 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.446098    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcmvl\" (UniqueName: \"kubernetes.io/projected/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-kube-api-access-fcmvl\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.446308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.446165    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.446308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.446219    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.446958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.446936    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.448765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.448745    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.454079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.454054    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcmvl\" (UniqueName: \"kubernetes.io/projected/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-kube-api-access-fcmvl\") pod \"maas-api-revoke-keys-e2e-api-a-151744-txq4m\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.538487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.538426    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:14.665269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.665178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:14.826717 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:47:14.826679    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1eadcb_b9da_451f_8c80_9423b7dd2ee9.slice/crio-47c0c29bc71db5c4c17a44789188e1c95828db381a15c0571100539b6d97d3b4 WatchSource:0}: Error finding container 47c0c29bc71db5c4c17a44789188e1c95828db381a15c0571100539b6d97d3b4: Status 404 returned error can't find the container with id 47c0c29bc71db5c4c17a44789188e1c95828db381a15c0571100539b6d97d3b4
Nov 03 10:47:14.829283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.829254    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"]
Nov 03 10:47:14.983475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.983441    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m" event={"ID":"df1eadcb-b9da-451f-8c80-9423b7dd2ee9","Type":"ContainerStarted","Data":"47c0c29bc71db5c4c17a44789188e1c95828db381a15c0571100539b6d97d3b4"}
Nov 03 10:47:14.985322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.985295    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk" event={"ID":"b5b651e1-3a26-4b8b-92b5-7f76a2aa1853","Type":"ContainerDied","Data":"25e460d8788a1b58e153f463b80743c66ba8849d3af3d2e8a8f6ecd67faae39f"}
Nov 03 10:47:14.985447 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.985350    5817 scope.go:122] "RemoveContainer" containerID="4503f8f2bd25ec34382c8cbbdde5ed0205fb54dcb1d64bb1ea714caab018e11c"
Nov 03 10:47:14.985508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.985474    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"
Nov 03 10:47:14.993866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.993841    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2" event={"ID":"29c3c4c4-6087-4653-9e86-8ce491859398","Type":"ContainerDied","Data":"a90d434f22a11f42dcfb43e27f605a9068da8f4facdff86600d5b82834074f0e"}
Nov 03 10:47:14.993991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:14.993868    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"
Nov 03 10:47:15.009805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.009786    5817 scope.go:122] "RemoveContainer" containerID="3a12cbcf8dc27588729020f07c178d9e60b2c9d062b394280af23b40468c7757"
Nov 03 10:47:15.035021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.034992    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"]
Nov 03 10:47:15.043961 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.043932    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-141053-6db8f5499f-vr6jk"]
Nov 03 10:47:15.048417 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.048396    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"]
Nov 03 10:47:15.052423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.052403    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-141053-6f49fbc9bc-vzzm2"]
Nov 03 10:47:15.547133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.547043    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:15.904564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.904529    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c3c4c4-6087-4653-9e86-8ce491859398" path="/var/lib/kubelet/pods/29c3c4c4-6087-4653-9e86-8ce491859398/volumes"
Nov 03 10:47:15.905096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.905078    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b651e1-3a26-4b8b-92b5-7f76a2aa1853" path="/var/lib/kubelet/pods/b5b651e1-3a26-4b8b-92b5-7f76a2aa1853/volumes"
Nov 03 10:47:15.994129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:15.994095    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:47:16.004339 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:16.004311    5817 generic.go:356] "Generic (PLEG): container finished" podID="df1eadcb-b9da-451f-8c80-9423b7dd2ee9" containerID="fe531d559d786c6dde61d4b700564bd300af979c98ba410081443fce094f9cbf" exitCode=0
Nov 03 10:47:16.004488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:16.004441    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m" event={"ID":"df1eadcb-b9da-451f-8c80-9423b7dd2ee9","Type":"ContainerDied","Data":"fe531d559d786c6dde61d4b700564bd300af979c98ba410081443fce094f9cbf"}
Nov 03 10:47:16.168557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:16.168468    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:17.209643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.209614    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:17.251363 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.251333    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-xks-ca\") pod \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") "
Nov 03 10:47:17.251544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.251376    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-openshift-ca\") pod \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") "
Nov 03 10:47:17.251544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.251454    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-kube-api-access-fcmvl\" (UniqueName: \"kubernetes.io/projected/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-kube-api-access-fcmvl\") pod \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\" (UID: \"df1eadcb-b9da-451f-8c80-9423b7dd2ee9\") "
Nov 03 10:47:17.251895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.251863    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-openshift-ca" pod "df1eadcb-b9da-451f-8c80-9423b7dd2ee9" (UID: "df1eadcb-b9da-451f-8c80-9423b7dd2ee9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:17.252427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.252391    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:17.254521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.254501    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-kube-api-access-fcmvl" pod "df1eadcb-b9da-451f-8c80-9423b7dd2ee9" (UID: "df1eadcb-b9da-451f-8c80-9423b7dd2ee9"). InnerVolumeSpecName "kube-api-access-fcmvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:17.254740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.254717    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-xks-ca" pod "df1eadcb-b9da-451f-8c80-9423b7dd2ee9" (UID: "df1eadcb-b9da-451f-8c80-9423b7dd2ee9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:47:17.354702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.354621    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:17.354702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.354656    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcmvl\" (UniqueName: \"kubernetes.io/projected/df1eadcb-b9da-451f-8c80-9423b7dd2ee9-kube-api-access-fcmvl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:17.520534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.520489    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:17.714633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:17.712931    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:18.032418 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:18.032342    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"
Nov 03 10:47:18.032558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:18.032344    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m" event={"ID":"df1eadcb-b9da-451f-8c80-9423b7dd2ee9","Type":"ContainerDied","Data":"47c0c29bc71db5c4c17a44789188e1c95828db381a15c0571100539b6d97d3b4"}
Nov 03 10:47:18.032558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:18.032471    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47c0c29bc71db5c4c17a44789188e1c95828db381a15c0571100539b6d97d3b4"
Nov 03 10:47:18.708360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:18.702095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:18.843943 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:18.843908    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:19.674149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:19.674116    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:20.370597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:20.366147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:21.025541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:21.025507    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"]
Nov 03 10:47:21.033066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:21.033035    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52692ea1-qfpz8"]
Nov 03 10:47:21.168437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:21.167106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:21.858608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:21.857292    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:21.911691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:21.910197    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e74d07d-656f-420e-997e-2b28d0aaade9" path="/var/lib/kubelet/pods/5e74d07d-656f-420e-997e-2b28d0aaade9/volumes"
Nov 03 10:47:22.385474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:22.385440    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:22.413095 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:47:22.413059    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:47:22.645649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:22.645072    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:22.835998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:22.835969    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:22.972432 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:22.972397    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"]
Nov 03 10:47:22.976789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:22.976768    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-141053-w8bng"]
Nov 03 10:47:23.132288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:23.132251    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:23.462648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:23.462616    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:23.906290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:23.906206    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96aa703c-e838-48c3-8dc7-185240898d94" path="/var/lib/kubelet/pods/96aa703c-e838-48c3-8dc7-185240898d94/volumes"
Nov 03 10:47:24.071334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:24.071274    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:24.343672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:24.343638    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"]
Nov 03 10:47:24.455957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:24.455915    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:24.592510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:24.592464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:25.144769 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:25.138545    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:25.747375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:25.747334    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:26.371626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:26.371559    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:26.514618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:26.514516    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:27.233300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:27.232996    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:27.908914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:27.908877    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:28.567604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:28.563656    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:29.022346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.022307    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"]
Nov 03 10:47:29.026761 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.026738    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe97acb-sdzkl"]
Nov 03 10:47:29.321125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.321038    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:29.344200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.344159    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l" podUID="7555ea1a-4b05-4c03-aed7-eefdb6ab5c80" containerName="payload-pre-processing" containerID="cri-o://0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b" gracePeriod=25
Nov 03 10:47:29.644954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.644927    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:47:29.655446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.655416    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:29.696599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.690263    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-plugins-config-volume\") pod \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\" (UID: \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\") "
Nov 03 10:47:29.696599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.690337    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-kube-api-access-tt5cz\" (UniqueName: \"kubernetes.io/projected/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-kube-api-access-tt5cz\") pod \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\" (UID: \"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80\") "
Nov 03 10:47:29.696599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.694804    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-plugins-config-volume" pod "7555ea1a-4b05-4c03-aed7-eefdb6ab5c80" (UID: "7555ea1a-4b05-4c03-aed7-eefdb6ab5c80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:29.701373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.701165    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:29.712694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.704138    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-kube-api-access-tt5cz" pod "7555ea1a-4b05-4c03-aed7-eefdb6ab5c80" (UID: "7555ea1a-4b05-4c03-aed7-eefdb6ab5c80"). InnerVolumeSpecName "kube-api-access-tt5cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:29.787767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.787731    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:29.802642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.802614    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5cz\" (UniqueName: \"kubernetes.io/projected/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80-kube-api-access-tt5cz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:29.906002 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:29.905922    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c88769-92c7-468f-965f-a790e4a17241" path="/var/lib/kubelet/pods/b0c88769-92c7-468f-965f-a790e4a17241/volumes"
Nov 03 10:47:30.055217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.054714    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:30.212614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.207071    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:30.222869 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.222833    5817 generic.go:356] "Generic (PLEG): container finished" podID="7555ea1a-4b05-4c03-aed7-eefdb6ab5c80" containerID="0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b" exitCode=0
Nov 03 10:47:30.223014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.222891    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"
Nov 03 10:47:30.223014 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.222919    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l" event={"ID":"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80","Type":"ContainerDied","Data":"0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b"}
Nov 03 10:47:30.223148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.223011    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l" event={"ID":"7555ea1a-4b05-4c03-aed7-eefdb6ab5c80","Type":"ContainerDied","Data":"7e92ec78351eca5741f4d9e702baeada5939c608f6982aec4737e25c941a999c"}
Nov 03 10:47:30.223148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.223041    5817 scope.go:122] "RemoveContainer" containerID="0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b"
Nov 03 10:47:30.237506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.237447    5817 scope.go:122] "RemoveContainer" containerID="0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b"
Nov 03 10:47:30.237824 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:47:30.237795    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b\": container with ID starting with 0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b not found: ID does not exist" containerID="0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b"
Nov 03 10:47:30.237944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.237836    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b"} err="failed to get container status \"0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b\": rpc error: code = NotFound desc = could not find container \"0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b\": container with ID starting with 0874790d01c66803cc5e6a38ddd746afc7553c001e91594311c0b063b5cceb6b not found: ID does not exist"
Nov 03 10:47:30.244055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.244034    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"]
Nov 03 10:47:30.248850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.248829    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-151744-5964df88ff-mf88l"]
Nov 03 10:47:30.328649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.328555    5817 scope.go:122] "RemoveContainer" containerID="f8079976cf57191b50a3d36fdd0e1695dff542912be4b5cb51c38669388fc6d6"
Nov 03 10:47:30.349846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.349637    5817 scope.go:122] "RemoveContainer" containerID="1bff1b19fb85fdbe38fc3ebb61ff999a12da6406b305a6b4117c62d14228cd77"
Nov 03 10:47:30.619467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.619398    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"]
Nov 03 10:47:30.620073 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.620059    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7555ea1a-4b05-4c03-aed7-eefdb6ab5c80" containerName="payload-pre-processing"
Nov 03 10:47:30.620115 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.620075    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7555ea1a-4b05-4c03-aed7-eefdb6ab5c80" containerName="payload-pre-processing"
Nov 03 10:47:30.620115 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.620091    5817 cpu_manager.go:414] "containerMap: removing container" podUID="df1eadcb-b9da-451f-8c80-9423b7dd2ee9" containerName="revoke-keys"
Nov 03 10:47:30.620115 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.620096    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1eadcb-b9da-451f-8c80-9423b7dd2ee9" containerName="revoke-keys"
Nov 03 10:47:30.620238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.620228    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1eadcb-b9da-451f-8c80-9423b7dd2ee9" containerName="revoke-keys"
Nov 03 10:47:30.620275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.620250    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7555ea1a-4b05-4c03-aed7-eefdb6ab5c80" containerName="payload-pre-processing"
Nov 03 10:47:30.626052 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.625977    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.629829 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.629788    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:47:30.630683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.630559    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:47:30.632249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.632227    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"]
Nov 03 10:47:30.743499 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.743462    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.743695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.743604    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bbk\" (UniqueName: \"kubernetes.io/projected/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-kube-api-access-b4bbk\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.743785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.743700    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.845730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.845698    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.845909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.845787    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bbk\" (UniqueName: \"kubernetes.io/projected/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-kube-api-access-b4bbk\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.845909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.845859    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.846437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.846419    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.848351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.848333    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.853068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.853051    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bbk\" (UniqueName: \"kubernetes.io/projected/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-kube-api-access-b4bbk\") pod \"maas-api-revoke-keys-e2e-api-b-a917b1-jwmml\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:30.944936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:30.944916    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:31.230783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:31.230751    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"]
Nov 03 10:47:31.232330 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:47:31.232302    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc6eb3d_9e0e_4373_ba7d_8cc6cc9cd665.slice/crio-e02757d6cd494b9f24b5c27fa504974085257fded4a56d2b4abf6e365b88a314 WatchSource:0}: Error finding container e02757d6cd494b9f24b5c27fa504974085257fded4a56d2b4abf6e365b88a314: Status 404 returned error can't find the container with id e02757d6cd494b9f24b5c27fa504974085257fded4a56d2b4abf6e365b88a314
Nov 03 10:47:31.435425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:31.435387    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:31.906597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:31.906510    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7555ea1a-4b05-4c03-aed7-eefdb6ab5c80" path="/var/lib/kubelet/pods/7555ea1a-4b05-4c03-aed7-eefdb6ab5c80/volumes"
Nov 03 10:47:32.245975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:32.245941    5817 generic.go:356] "Generic (PLEG): container finished" podID="bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665" containerID="c0e52f2b712ce87a6089cca429d91ee71ea08d9d7f799353bb5509c766935c1a" exitCode=0
Nov 03 10:47:32.246358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:32.246034    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml" event={"ID":"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665","Type":"ContainerDied","Data":"c0e52f2b712ce87a6089cca429d91ee71ea08d9d7f799353bb5509c766935c1a"}
Nov 03 10:47:32.246358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:32.246072    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml" event={"ID":"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665","Type":"ContainerStarted","Data":"e02757d6cd494b9f24b5c27fa504974085257fded4a56d2b4abf6e365b88a314"}
Nov 03 10:47:32.459671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:32.459215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:33.169362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.169324    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:33.423805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.423746    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:33.499042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.499007    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-kube-api-access-b4bbk\" (UniqueName: \"kubernetes.io/projected/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-kube-api-access-b4bbk\") pod \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") "
Nov 03 10:47:33.499228 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.499099    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-openshift-ca\") pod \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") "
Nov 03 10:47:33.499501 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.499476    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-openshift-ca" pod "bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665" (UID: "bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:33.499906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.499842    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-xks-ca\") pod \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\" (UID: \"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665\") "
Nov 03 10:47:33.500934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.500912    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:33.502369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.502334    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-xks-ca" pod "bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665" (UID: "bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:47:33.502701 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.502674    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-kube-api-access-b4bbk" pod "bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665" (UID: "bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665"). InnerVolumeSpecName "kube-api-access-b4bbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:33.603142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.603111    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:33.603142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.603142    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4bbk\" (UniqueName: \"kubernetes.io/projected/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665-kube-api-access-b4bbk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:33.993853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:33.993808    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:34.027994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.027956    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"]
Nov 03 10:47:34.035018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.034995    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11264373-gvmd2"]
Nov 03 10:47:34.265779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.265746    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml" event={"ID":"bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665","Type":"ContainerDied","Data":"e02757d6cd494b9f24b5c27fa504974085257fded4a56d2b4abf6e365b88a314"}
Nov 03 10:47:34.265779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.265777    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"
Nov 03 10:47:34.266003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.265802    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e02757d6cd494b9f24b5c27fa504974085257fded4a56d2b4abf6e365b88a314"
Nov 03 10:47:34.379564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.379536    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"]
Nov 03 10:47:34.380221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.380207    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665" containerName="revoke-keys"
Nov 03 10:47:34.380261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.380222    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665" containerName="revoke-keys"
Nov 03 10:47:34.380342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.380333    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665" containerName="revoke-keys"
Nov 03 10:47:34.384156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.384139    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.388384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.388279    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:47:34.388384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.388343    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:47:34.392877 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.392855    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"]
Nov 03 10:47:34.518214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.518124    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.518214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.518162    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgd72\" (UniqueName: \"kubernetes.io/projected/12a01722-f51b-4932-9f6f-81511daefeb4-kube-api-access-bgd72\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.518214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.518195    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.620394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.620349    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.620626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.620404    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgd72\" (UniqueName: \"kubernetes.io/projected/12a01722-f51b-4932-9f6f-81511daefeb4-kube-api-access-bgd72\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.620626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.620452    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.621176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.621147    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.623250 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.623221    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.628109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.628082    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgd72\" (UniqueName: \"kubernetes.io/projected/12a01722-f51b-4932-9f6f-81511daefeb4-kube-api-access-bgd72\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-zv8d6\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.706650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.706623    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:34.995615 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:47:34.995566    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a01722_f51b_4932_9f6f_81511daefeb4.slice/crio-7555c21cfeba8cc70a1d632107a696cc7da7b6b75ff18d27f24d00e89a54e60c WatchSource:0}: Error finding container 7555c21cfeba8cc70a1d632107a696cc7da7b6b75ff18d27f24d00e89a54e60c: Status 404 returned error can't find the container with id 7555c21cfeba8cc70a1d632107a696cc7da7b6b75ff18d27f24d00e89a54e60c
Nov 03 10:47:34.996965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:34.996941    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"]
Nov 03 10:47:35.145651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:35.141990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:35.275307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:35.275275    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6" event={"ID":"12a01722-f51b-4932-9f6f-81511daefeb4","Type":"ContainerStarted","Data":"7555c21cfeba8cc70a1d632107a696cc7da7b6b75ff18d27f24d00e89a54e60c"}
Nov 03 10:47:35.770041 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:35.770011    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:35.904622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:35.904548    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f658ba-4bbb-4ffd-9e31-e614e74f8e38" path="/var/lib/kubelet/pods/13f658ba-4bbb-4ffd-9e31-e614e74f8e38/volumes"
Nov 03 10:47:36.322850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:36.315056    5817 generic.go:356] "Generic (PLEG): container finished" podID="12a01722-f51b-4932-9f6f-81511daefeb4" containerID="03e0161c9305dfec25e3caeb2c17ac4cbe9ab33298fd91eb15d85d315fdf9656" exitCode=0
Nov 03 10:47:36.322850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:36.315148    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6" event={"ID":"12a01722-f51b-4932-9f6f-81511daefeb4","Type":"ContainerDied","Data":"03e0161c9305dfec25e3caeb2c17ac4cbe9ab33298fd91eb15d85d315fdf9656"}
Nov 03 10:47:36.721602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:36.720550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:37.387963 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.387927    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:37.535880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.535854    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:37.664374 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.664036    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:37.681356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.681307    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-openshift-ca\") pod \"12a01722-f51b-4932-9f6f-81511daefeb4\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") "
Nov 03 10:47:37.681557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.681524    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a01722-f51b-4932-9f6f-81511daefeb4-kube-api-access-bgd72\" (UniqueName: \"kubernetes.io/projected/12a01722-f51b-4932-9f6f-81511daefeb4-kube-api-access-bgd72\") pod \"12a01722-f51b-4932-9f6f-81511daefeb4\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") "
Nov 03 10:47:37.681655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.681614    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-xks-ca\") pod \"12a01722-f51b-4932-9f6f-81511daefeb4\" (UID: \"12a01722-f51b-4932-9f6f-81511daefeb4\") "
Nov 03 10:47:37.682203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.682170    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-openshift-ca" pod "12a01722-f51b-4932-9f6f-81511daefeb4" (UID: "12a01722-f51b-4932-9f6f-81511daefeb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:37.682423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.682361    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:37.684976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.684947    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-xks-ca" pod "12a01722-f51b-4932-9f6f-81511daefeb4" (UID: "12a01722-f51b-4932-9f6f-81511daefeb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:47:37.702045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.685961    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a01722-f51b-4932-9f6f-81511daefeb4-kube-api-access-bgd72" pod "12a01722-f51b-4932-9f6f-81511daefeb4" (UID: "12a01722-f51b-4932-9f6f-81511daefeb4"). InnerVolumeSpecName "kube-api-access-bgd72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:37.784118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.784085    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a01722-f51b-4932-9f6f-81511daefeb4-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:37.784118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:37.784120    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgd72\" (UniqueName: \"kubernetes.io/projected/12a01722-f51b-4932-9f6f-81511daefeb4-kube-api-access-bgd72\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:38.339734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:38.339705    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"
Nov 03 10:47:38.339916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:38.339746    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6" event={"ID":"12a01722-f51b-4932-9f6f-81511daefeb4","Type":"ContainerDied","Data":"7555c21cfeba8cc70a1d632107a696cc7da7b6b75ff18d27f24d00e89a54e60c"}
Nov 03 10:47:38.339916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:38.339797    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7555c21cfeba8cc70a1d632107a696cc7da7b6b75ff18d27f24d00e89a54e60c"
Nov 03 10:47:38.411372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:38.411280    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:39.040423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:39.040387    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:39.574640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:39.573655    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:39.712954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:39.712909    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"]
Nov 03 10:47:39.719609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:39.717897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:39.719609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:39.718095    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-151744-txq4m"]
Nov 03 10:47:39.986542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:39.986497    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1eadcb-b9da-451f-8c80-9423b7dd2ee9" path="/var/lib/kubelet/pods/df1eadcb-b9da-451f-8c80-9423b7dd2ee9/volumes"
Nov 03 10:47:40.218353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:40.217510    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:40.353466 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:40.353381    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:40.741632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:40.738259    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"]
Nov 03 10:47:40.758593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:40.751602    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"]
Nov 03 10:47:40.855622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:40.850466    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:40.981490 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:40.981455    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:41.127685 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:41.124651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:41.460446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:41.460414    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:41.764362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:41.762432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:42.336604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:42.336372    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:42.957441 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:42.957404    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:43.036594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:43.035657    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"]
Nov 03 10:47:43.042380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:43.042344    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e520a03a-bgbx5"]
Nov 03 10:47:43.542451 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:43.540655    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:43.926717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:43.926633    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e877ec-de2f-4359-ac50-925a16eabffe" path="/var/lib/kubelet/pods/06e877ec-de2f-4359-ac50-925a16eabffe/volumes"
Nov 03 10:47:44.241956 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:44.241847    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:44.658293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:44.658215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:44.814940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:44.812204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:45.335070 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:45.334191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:45.740625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:45.740123    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5" podUID="e0ae0509-f41d-4eb4-aa40-2934662640c7" containerName="payload-processing" containerID="cri-o://7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d" gracePeriod=25
Nov 03 10:47:45.752898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:45.752873    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg" podUID="08f290f8-ff27-49df-bc59-afefbe16608d" containerName="payload-pre-processing" containerID="cri-o://628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8" gracePeriod=25
Nov 03 10:47:46.006548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.006459    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:46.159486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.159423    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:47:46.183121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.183083    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ae0509-f41d-4eb4-aa40-2934662640c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ae0509-f41d-4eb4-aa40-2934662640c7-plugins-config-volume\") pod \"e0ae0509-f41d-4eb4-aa40-2934662640c7\" (UID: \"e0ae0509-f41d-4eb4-aa40-2934662640c7\") "
Nov 03 10:47:46.183274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.183196    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ae0509-f41d-4eb4-aa40-2934662640c7-kube-api-access-ndt5j\" (UniqueName: \"kubernetes.io/projected/e0ae0509-f41d-4eb4-aa40-2934662640c7-kube-api-access-ndt5j\") pod \"e0ae0509-f41d-4eb4-aa40-2934662640c7\" (UID: \"e0ae0509-f41d-4eb4-aa40-2934662640c7\") "
Nov 03 10:47:46.211772 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.184672    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ae0509-f41d-4eb4-aa40-2934662640c7-plugins-config-volume" pod "e0ae0509-f41d-4eb4-aa40-2934662640c7" (UID: "e0ae0509-f41d-4eb4-aa40-2934662640c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:46.237653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.237507    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ae0509-f41d-4eb4-aa40-2934662640c7-kube-api-access-ndt5j" pod "e0ae0509-f41d-4eb4-aa40-2934662640c7" (UID: "e0ae0509-f41d-4eb4-aa40-2934662640c7"). InnerVolumeSpecName "kube-api-access-ndt5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:46.277311 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.277287    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:47:46.285234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.285200    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndt5j\" (UniqueName: \"kubernetes.io/projected/e0ae0509-f41d-4eb4-aa40-2934662640c7-kube-api-access-ndt5j\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:46.285234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.285234    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ae0509-f41d-4eb4-aa40-2934662640c7-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:46.386487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.386449    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f290f8-ff27-49df-bc59-afefbe16608d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f290f8-ff27-49df-bc59-afefbe16608d-plugins-config-volume\") pod \"08f290f8-ff27-49df-bc59-afefbe16608d\" (UID: \"08f290f8-ff27-49df-bc59-afefbe16608d\") "
Nov 03 10:47:46.386987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.386509    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f290f8-ff27-49df-bc59-afefbe16608d-kube-api-access-d44ct\" (UniqueName: \"kubernetes.io/projected/08f290f8-ff27-49df-bc59-afefbe16608d-kube-api-access-d44ct\") pod \"08f290f8-ff27-49df-bc59-afefbe16608d\" (UID: \"08f290f8-ff27-49df-bc59-afefbe16608d\") "
Nov 03 10:47:46.386987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.386923    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f290f8-ff27-49df-bc59-afefbe16608d-plugins-config-volume" pod "08f290f8-ff27-49df-bc59-afefbe16608d" (UID: "08f290f8-ff27-49df-bc59-afefbe16608d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:46.387318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.387295    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f290f8-ff27-49df-bc59-afefbe16608d-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:46.389370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.389341    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f290f8-ff27-49df-bc59-afefbe16608d-kube-api-access-d44ct" pod "08f290f8-ff27-49df-bc59-afefbe16608d" (UID: "08f290f8-ff27-49df-bc59-afefbe16608d"). InnerVolumeSpecName "kube-api-access-d44ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:46.435092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.435041    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:46.471813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.471778    5817 generic.go:356] "Generic (PLEG): container finished" podID="e0ae0509-f41d-4eb4-aa40-2934662640c7" containerID="7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d" exitCode=0
Nov 03 10:47:46.472001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.471840    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"
Nov 03 10:47:46.472001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.471860    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5" event={"ID":"e0ae0509-f41d-4eb4-aa40-2934662640c7","Type":"ContainerDied","Data":"7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d"}
Nov 03 10:47:46.472001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.471922    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5" event={"ID":"e0ae0509-f41d-4eb4-aa40-2934662640c7","Type":"ContainerDied","Data":"0704e758918ff60f84b4d556134628e2b3b273de685d0f376f368ccd27237d06"}
Nov 03 10:47:46.472001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.471953    5817 scope.go:122] "RemoveContainer" containerID="7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d"
Nov 03 10:47:46.473848 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.473829    5817 generic.go:356] "Generic (PLEG): container finished" podID="08f290f8-ff27-49df-bc59-afefbe16608d" containerID="628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8" exitCode=0
Nov 03 10:47:46.473955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.473929    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg" event={"ID":"08f290f8-ff27-49df-bc59-afefbe16608d","Type":"ContainerDied","Data":"628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8"}
Nov 03 10:47:46.474011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.473955    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg" event={"ID":"08f290f8-ff27-49df-bc59-afefbe16608d","Type":"ContainerDied","Data":"a6277c85e776bc38baed540b12bc3e46afc074501a194da0863190690d371f39"}
Nov 03 10:47:46.474011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.474008    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"
Nov 03 10:47:46.489083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.488991    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44ct\" (UniqueName: \"kubernetes.io/projected/08f290f8-ff27-49df-bc59-afefbe16608d-kube-api-access-d44ct\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:46.489083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.489064    5817 scope.go:122] "RemoveContainer" containerID="7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d"
Nov 03 10:47:46.489468 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:47:46.489418    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d\": container with ID starting with 7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d not found: ID does not exist" containerID="7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d"
Nov 03 10:47:46.489468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.489455    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d"} err="failed to get container status \"7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d\": rpc error: code = NotFound desc = could not find container \"7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d\": container with ID starting with 7ff50d8788234f687bced45d3a070318abf487bf3b09dceb9a879233bec47c3d not found: ID does not exist"
Nov 03 10:47:46.489688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.489477    5817 scope.go:122] "RemoveContainer" containerID="628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8"
Nov 03 10:47:46.507827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.507798    5817 scope.go:122] "RemoveContainer" containerID="628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8"
Nov 03 10:47:46.509305 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:47:46.509272    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8\": container with ID starting with 628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8 not found: ID does not exist" containerID="628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8"
Nov 03 10:47:46.509427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.509314    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8"} err="failed to get container status \"628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8\": rpc error: code = NotFound desc = could not find container \"628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8\": container with ID starting with 628cc0f869379dd1d150ab33e52106cfa96cb30f73883679d28367547e9ac2f8 not found: ID does not exist"
Nov 03 10:47:46.512788 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.512683    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"]
Nov 03 10:47:46.519629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.519547    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a917b1-87b985f47-4kdl5"]
Nov 03 10:47:46.533129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.532749    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"]
Nov 03 10:47:46.553854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.553634    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a917b1-869f8fb44c-4ntmg"]
Nov 03 10:47:46.564088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.564059    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:46.742637 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.741184    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"]
Nov 03 10:47:46.742637 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.741416    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy" containerID="cri-o://5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515" gracePeriod=30
Nov 03 10:47:46.779045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.778883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:46.926700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:46.917311    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:47.250961 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.250931    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"]
Nov 03 10:47:47.251740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.251720    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e0ae0509-f41d-4eb4-aa40-2934662640c7" containerName="payload-processing"
Nov 03 10:47:47.251834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.251744    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ae0509-f41d-4eb4-aa40-2934662640c7" containerName="payload-processing"
Nov 03 10:47:47.251834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.251779    5817 cpu_manager.go:414] "containerMap: removing container" podUID="08f290f8-ff27-49df-bc59-afefbe16608d" containerName="payload-pre-processing"
Nov 03 10:47:47.251834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.251787    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f290f8-ff27-49df-bc59-afefbe16608d" containerName="payload-pre-processing"
Nov 03 10:47:47.251834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.251821    5817 cpu_manager.go:414] "containerMap: removing container" podUID="12a01722-f51b-4932-9f6f-81511daefeb4" containerName="revoke-keys"
Nov 03 10:47:47.251834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.251828    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a01722-f51b-4932-9f6f-81511daefeb4" containerName="revoke-keys"
Nov 03 10:47:47.252016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.251985    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a01722-f51b-4932-9f6f-81511daefeb4" containerName="revoke-keys"
Nov 03 10:47:47.252058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.252016    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ae0509-f41d-4eb4-aa40-2934662640c7" containerName="payload-processing"
Nov 03 10:47:47.252058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.252027    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f290f8-ff27-49df-bc59-afefbe16608d" containerName="payload-pre-processing"
Nov 03 10:47:47.260903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.260880    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.261136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.261109    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"]
Nov 03 10:47:47.281157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.281119    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:47:47.281973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.281937    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:47:47.330676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.330637    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkxz\" (UniqueName: \"kubernetes.io/projected/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-kube-api-access-jqkxz\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.330847 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.330761    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.331036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.331015    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.433168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.433128    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.433678 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.433299    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkxz\" (UniqueName: \"kubernetes.io/projected/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-kube-api-access-jqkxz\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.433678 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.433347    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.434184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.434160    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.436219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.436194    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.441252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.441226    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkxz\" (UniqueName: \"kubernetes.io/projected/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-kube-api-access-jqkxz\") pod \"maas-api-revoke-keys-e2e-api-b-273012-zqbcr\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.585479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.585399    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:47.775595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.756199    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:47.886497 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:47:47.886464    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643de369_2c0b_4ef9_ae0b_d87cda35ba3b.slice/crio-886b0ce9f0e9126fd9b38b131a06e46efc60aa548d4649dde313034a962004eb WatchSource:0}: Error finding container 886b0ce9f0e9126fd9b38b131a06e46efc60aa548d4649dde313034a962004eb: Status 404 returned error can't find the container with id 886b0ce9f0e9126fd9b38b131a06e46efc60aa548d4649dde313034a962004eb
Nov 03 10:47:47.888092 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.888062    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"]
Nov 03 10:47:47.912328 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.909454    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f290f8-ff27-49df-bc59-afefbe16608d" path="/var/lib/kubelet/pods/08f290f8-ff27-49df-bc59-afefbe16608d/volumes"
Nov 03 10:47:47.912328 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:47.909939    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ae0509-f41d-4eb4-aa40-2934662640c7" path="/var/lib/kubelet/pods/e0ae0509-f41d-4eb4-aa40-2934662640c7/volumes"
Nov 03 10:47:48.022204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:48.022157    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"]
Nov 03 10:47:48.026940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:48.026913    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc69b3fa-xxvnd"]
Nov 03 10:47:48.455944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:48.454928    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:48.513370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:48.513281    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr" event={"ID":"643de369-2c0b-4ef9-ae0b-d87cda35ba3b","Type":"ContainerStarted","Data":"ae31e1d6f825bde5b032f611c81f5e075fd00d8cc452296fbc51223a8be6fab5"}
Nov 03 10:47:48.513370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:48.513355    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr" event={"ID":"643de369-2c0b-4ef9-ae0b-d87cda35ba3b","Type":"ContainerStarted","Data":"886b0ce9f0e9126fd9b38b131a06e46efc60aa548d4649dde313034a962004eb"}
Nov 03 10:47:48.541760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:48.541693    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr" podStartSLOduration=1.5416767770000002 podStartE2EDuration="1.541676777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:48.533468413 +0000 UTC m=+1098.739050465" watchObservedRunningTime="2026-11-03 10:47:48.541676777 +0000 UTC m=+1098.747258830"
Nov 03 10:47:49.416697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.416595    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:49.548151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.548119    5817 generic.go:356] "Generic (PLEG): container finished" podID="643de369-2c0b-4ef9-ae0b-d87cda35ba3b" containerID="ae31e1d6f825bde5b032f611c81f5e075fd00d8cc452296fbc51223a8be6fab5" exitCode=0
Nov 03 10:47:49.548538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.548203    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr" event={"ID":"643de369-2c0b-4ef9-ae0b-d87cda35ba3b","Type":"ContainerDied","Data":"ae31e1d6f825bde5b032f611c81f5e075fd00d8cc452296fbc51223a8be6fab5"}
Nov 03 10:47:49.858037 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.858003    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"]
Nov 03 10:47:49.881082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.881049    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"]
Nov 03 10:47:49.881239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.881101    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:49.911473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.911443    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27b86fd-0819-4084-bfa1-f99dbd2d5884" path="/var/lib/kubelet/pods/f27b86fd-0819-4084-bfa1-f99dbd2d5884/volumes"
Nov 03 10:47:49.977194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.977157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:49.983761 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.983741    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:49.983878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.983786    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtqd\" (UniqueName: \"kubernetes.io/projected/427a43ca-e3f1-4616-be50-352afca82a2f-kube-api-access-vmtqd\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:49.983926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:49.983888    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:50.085327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.085295    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtqd\" (UniqueName: \"kubernetes.io/projected/427a43ca-e3f1-4616-be50-352afca82a2f-kube-api-access-vmtqd\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:50.085492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.085369    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:50.085677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.085657    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:50.086266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.086188    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:50.088343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.088314    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:50.093303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.093271    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtqd\" (UniqueName: \"kubernetes.io/projected/427a43ca-e3f1-4616-be50-352afca82a2f-kube-api-access-vmtqd\") pod \"maas-api-revoke-sub-1f0956c4-pnvfx\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:50.206595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.206076    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:50.222612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.221021    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:50.453058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.453026    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"]
Nov 03 10:47:50.458597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.458519    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a917b1-jwmml"]
Nov 03 10:47:50.489212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.489184    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:50.643594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.633532    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:50.672627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.664848    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"]
Nov 03 10:47:50.999068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:50.999033    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:51.043417 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.043389    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:51.078626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.078534    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-openshift-ca\") pod \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") "
Nov 03 10:47:51.078766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.078709    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-kube-api-access-jqkxz\" (UniqueName: \"kubernetes.io/projected/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-kube-api-access-jqkxz\") pod \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") "
Nov 03 10:47:51.078823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.078789    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-xks-ca\") pod \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\" (UID: \"643de369-2c0b-4ef9-ae0b-d87cda35ba3b\") "
Nov 03 10:47:51.080459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.079011    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-openshift-ca" pod "643de369-2c0b-4ef9-ae0b-d87cda35ba3b" (UID: "643de369-2c0b-4ef9-ae0b-d87cda35ba3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:51.080459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.079792    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:51.103039 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.081167    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-xks-ca" pod "643de369-2c0b-4ef9-ae0b-d87cda35ba3b" (UID: "643de369-2c0b-4ef9-ae0b-d87cda35ba3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:47:51.103039 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.081343    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-kube-api-access-jqkxz" pod "643de369-2c0b-4ef9-ae0b-d87cda35ba3b" (UID: "643de369-2c0b-4ef9-ae0b-d87cda35ba3b"). InnerVolumeSpecName "kube-api-access-jqkxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:51.181209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.181172    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqkxz\" (UniqueName: \"kubernetes.io/projected/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-kube-api-access-jqkxz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:51.181209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.181207    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643de369-2c0b-4ef9-ae0b-d87cda35ba3b-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:51.316259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.316224    5817 patch_prober.go:29] interesting pod/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:47:51.316392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.316283    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:47:51.547190 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.542610    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:51.580230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.580191    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx" event={"ID":"427a43ca-e3f1-4616-be50-352afca82a2f","Type":"ContainerStarted","Data":"7ca4f3a878b96fd10c4bdea1c0c86c466f1f040d72b5f342c064849047d458ca"}
Nov 03 10:47:51.580515 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.580244    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx" event={"ID":"427a43ca-e3f1-4616-be50-352afca82a2f","Type":"ContainerStarted","Data":"ebc566a9a4da12493988d042b85e3d40719b5936f01e36930b7cc8cc3196d94d"}
Nov 03 10:47:51.583687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.583662    5817 generic.go:356] "Generic (PLEG): container finished" podID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerID="2ae754ea1635d766e917c64d1f1daf6188d5535d8d242441ab9348e41627cd34" exitCode=0
Nov 03 10:47:51.583849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.583738    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" event={"ID":"0d39912f-e2d5-4961-a3f4-42d08009885a","Type":"ContainerDied","Data":"2ae754ea1635d766e917c64d1f1daf6188d5535d8d242441ab9348e41627cd34"}
Nov 03 10:47:51.584293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.584275    5817 scope.go:122] "RemoveContainer" containerID="2ae754ea1635d766e917c64d1f1daf6188d5535d8d242441ab9348e41627cd34"
Nov 03 10:47:51.586117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.586086    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:51.586217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.586156    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:51.586298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.586282    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:51.587660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.587629    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"
Nov 03 10:47:51.587765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.587675    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr" event={"ID":"643de369-2c0b-4ef9-ae0b-d87cda35ba3b","Type":"ContainerDied","Data":"886b0ce9f0e9126fd9b38b131a06e46efc60aa548d4649dde313034a962004eb"}
Nov 03 10:47:51.587765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.587716    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886b0ce9f0e9126fd9b38b131a06e46efc60aa548d4649dde313034a962004eb"
Nov 03 10:47:51.602146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.602097    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx" podStartSLOduration=2.602082291 podStartE2EDuration="2.602082291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:51.595997995 +0000 UTC m=+1101.801580049" watchObservedRunningTime="2026-11-03 10:47:51.602082291 +0000 UTC m=+1101.807664343"
Nov 03 10:47:51.836934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.836869    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:51.910971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:51.910512    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665" path="/var/lib/kubelet/pods/bcc6eb3d-9e0e-4373-ba7d-8cc6cc9cd665/volumes"
Nov 03 10:47:52.071258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.070950    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:47:52.220598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220124    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-envoy\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.220598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220192    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-credential-socket\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.220598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220247    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-certs\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.220598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220367    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-token\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.220598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220420    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-kube-api-access-h8v58\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-kube-api-access-h8v58\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.220598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220455    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-podinfo\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.220598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220493    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-data\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.221138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220649    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-socket\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.221138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.220895    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe18144-9897-4034-99ff-f5ddec642f0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18144-9897-4034-99ff-f5ddec642f0d-istiod-ca-cert\") pod \"ffe18144-9897-4034-99ff-f5ddec642f0d\" (UID: \"ffe18144-9897-4034-99ff-f5ddec642f0d\") "
Nov 03 10:47:52.221548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.221340    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-credential-socket" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:52.221713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.221692    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe18144-9897-4034-99ff-f5ddec642f0d-istiod-ca-cert" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:52.222178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.221973    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.222178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.221999    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe18144-9897-4034-99ff-f5ddec642f0d-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.222549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.222387    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-certs" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:52.222709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.222679    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-data" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:52.222860 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.222834    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-socket" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:52.225208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.225169    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-kube-api-access-h8v58" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "kube-api-access-h8v58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:52.225644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.225622    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-podinfo" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:47:52.226309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.226288    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-envoy" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:52.226769 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.226736    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-token" pod "ffe18144-9897-4034-99ff-f5ddec642f0d" (UID: "ffe18144-9897-4034-99ff-f5ddec642f0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:52.250300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.250176    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:52.330595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.326301    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.330595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.326352    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8v58\" (UniqueName: \"kubernetes.io/projected/ffe18144-9897-4034-99ff-f5ddec642f0d-kube-api-access-h8v58\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.330595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.326370    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.330595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.326385    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.330595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.326415    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.330595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.326430    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.330595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.326443    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe18144-9897-4034-99ff-f5ddec642f0d-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.436309 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:47:52.436250    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:47:52.609642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.609527    5817 generic.go:356] "Generic (PLEG): container finished" podID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerID="5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515" exitCode=0
Nov 03 10:47:52.609985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.609958    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" event={"ID":"ffe18144-9897-4034-99ff-f5ddec642f0d","Type":"ContainerDied","Data":"5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515"}
Nov 03 10:47:52.610157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.610138    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm" event={"ID":"ffe18144-9897-4034-99ff-f5ddec642f0d","Type":"ContainerDied","Data":"145a171c49176af6057f48cb055a8acd90961404b3a4e32052c2d4df5de7175a"}
Nov 03 10:47:52.610285 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.610271    5817 scope.go:122] "RemoveContainer" containerID="5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515"
Nov 03 10:47:52.610540 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.610521    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"
Nov 03 10:47:52.620905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.620873    5817 generic.go:356] "Generic (PLEG): container finished" podID="427a43ca-e3f1-4616-be50-352afca82a2f" containerID="7ca4f3a878b96fd10c4bdea1c0c86c466f1f040d72b5f342c064849047d458ca" exitCode=0
Nov 03 10:47:52.621007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.620994    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx" event={"ID":"427a43ca-e3f1-4616-be50-352afca82a2f","Type":"ContainerDied","Data":"7ca4f3a878b96fd10c4bdea1c0c86c466f1f040d72b5f342c064849047d458ca"}
Nov 03 10:47:52.624259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.624237    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" event={"ID":"0d39912f-e2d5-4961-a3f4-42d08009885a","Type":"ContainerStarted","Data":"9e457413c4f48ea3773b0713c065885b9e358955dcc7663d7d8ee2d40a563aaa"}
Nov 03 10:47:52.632502 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.632420    5817 scope.go:122] "RemoveContainer" containerID="b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e"
Nov 03 10:47:52.646546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.646533    5817 scope.go:122] "RemoveContainer" containerID="5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515"
Nov 03 10:47:52.646890 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:47:52.646867    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515\": container with ID starting with 5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515 not found: ID does not exist" containerID="5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515"
Nov 03 10:47:52.646987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.646894    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515"} err="failed to get container status \"5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515\": rpc error: code = NotFound desc = could not find container \"5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515\": container with ID starting with 5ce2480907c5068d6bd703b5185a4bd1a82e4971881f0ed92a36580448a64515 not found: ID does not exist"
Nov 03 10:47:52.646987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.646910    5817 scope.go:122] "RemoveContainer" containerID="b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e"
Nov 03 10:47:52.647181 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:47:52.647161    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e\": container with ID starting with b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e not found: ID does not exist" containerID="b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e"
Nov 03 10:47:52.647243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.647188    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e"} err="failed to get container status \"b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e\": rpc error: code = NotFound desc = could not find container \"b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e\": container with ID starting with b8239d92e04851a865e68c253254c5c70e71e984f858b7a66ecb1de3f9febf3e not found: ID does not exist"
Nov 03 10:47:52.676914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.676885    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"]
Nov 03 10:47:52.681143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.681120    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-151744-openshift-default-68fc44b48f-dnvwm"]
Nov 03 10:47:52.861373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.861261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:52.872028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.871944    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:52.874587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.873952    5817 patch_prober.go:29] interesting pod/e2e-api-b-642fe5-openshift-default-76fb996475-967bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 10:47:52.874587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:52.874021    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 10:47:53.461121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:53.461079    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:53.858471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:53.858448    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:53.871788 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:53.871761    5817 patch_prober.go:29] interesting pod/e2e-api-b-642fe5-openshift-default-76fb996475-967bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 10:47:53.872147 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:53.871805    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 10:47:53.934711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:53.934152    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" path="/var/lib/kubelet/pods/ffe18144-9897-4034-99ff-f5ddec642f0d/volumes"
Nov 03 10:47:54.024099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.024066    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"]
Nov 03 10:47:54.028396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.028370    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5bda6a-587qp"]
Nov 03 10:47:54.051875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.051837    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-xks-ca\") pod \"427a43ca-e3f1-4616-be50-352afca82a2f\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") "
Nov 03 10:47:54.052091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.051916    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427a43ca-e3f1-4616-be50-352afca82a2f-kube-api-access-vmtqd\" (UniqueName: \"kubernetes.io/projected/427a43ca-e3f1-4616-be50-352afca82a2f-kube-api-access-vmtqd\") pod \"427a43ca-e3f1-4616-be50-352afca82a2f\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") "
Nov 03 10:47:54.052091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.052046    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-openshift-ca\") pod \"427a43ca-e3f1-4616-be50-352afca82a2f\" (UID: \"427a43ca-e3f1-4616-be50-352afca82a2f\") "
Nov 03 10:47:54.056517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.056440    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-openshift-ca" pod "427a43ca-e3f1-4616-be50-352afca82a2f" (UID: "427a43ca-e3f1-4616-be50-352afca82a2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:54.056887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.056858    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:54.059470 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.058777    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:54.065043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.062083    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-xks-ca" pod "427a43ca-e3f1-4616-be50-352afca82a2f" (UID: "427a43ca-e3f1-4616-be50-352afca82a2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:47:54.069509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.069459    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427a43ca-e3f1-4616-be50-352afca82a2f-kube-api-access-vmtqd" pod "427a43ca-e3f1-4616-be50-352afca82a2f" (UID: "427a43ca-e3f1-4616-be50-352afca82a2f"). InnerVolumeSpecName "kube-api-access-vmtqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:54.158124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.158092    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427a43ca-e3f1-4616-be50-352afca82a2f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:54.158124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.158121    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmtqd\" (UniqueName: \"kubernetes.io/projected/427a43ca-e3f1-4616-be50-352afca82a2f-kube-api-access-vmtqd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:54.657639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.656931    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:54.680827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.679769    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"
Nov 03 10:47:54.705610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.704667    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx" event={"ID":"427a43ca-e3f1-4616-be50-352afca82a2f","Type":"ContainerDied","Data":"ebc566a9a4da12493988d042b85e3d40719b5936f01e36930b7cc8cc3196d94d"}
Nov 03 10:47:54.705610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.704785    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebc566a9a4da12493988d042b85e3d40719b5936f01e36930b7cc8cc3196d94d"
Nov 03 10:47:54.871034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.871002    5817 patch_prober.go:29] interesting pod/e2e-api-b-642fe5-openshift-default-76fb996475-967bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 10:47:54.871203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:54.871051    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 10:47:55.266673 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:55.266635    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:55.871104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:55.871076    5817 patch_prober.go:29] interesting pod/e2e-api-b-642fe5-openshift-default-76fb996475-967bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 10:47:55.871274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:55.871122    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 10:47:55.922779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:55.922746    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd5b757-fe1a-4755-8adf-30e8d630da0e" path="/var/lib/kubelet/pods/6fd5b757-fe1a-4755-8adf-30e8d630da0e/volumes"
Nov 03 10:47:56.045921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:56.045883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:56.657028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:56.656990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:56.871236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:56.871201    5817 patch_prober.go:29] interesting pod/e2e-api-b-642fe5-openshift-default-76fb996475-967bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 10:47:56.871436 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:56.871267    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 10:47:56.976480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:56.976447    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:57.122598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:57.118522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:57.360183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:57.360075    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:57.703713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:57.703670    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:57.928804 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:57.928763    5817 patch_prober.go:29] interesting pod/e2e-api-b-642fe5-openshift-default-76fb996475-967bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 10:47:57.929103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:57.929056    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 10:47:58.014408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:57.958087    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:58.163226 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:58.163185    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:58.675743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:58.675710    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:58.825922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:58.825073    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:58.884711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:58.881804    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:58.884711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:58.883664    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:58.900243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:58.887861    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:47:59.004594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:59.004090    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:47:59.853762 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:47:59.853723    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:00.643975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:00.638355    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:01.341612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:01.337860    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:02.048966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:02.048921    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:02.780423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:02.780281    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:03.450361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:03.450321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:03.775172 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:03.775097    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:48:04.285003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:04.284965    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:05.087191 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:05.087160    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:05.766865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:05.766831    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:06.448299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:06.448265    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:07.052754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:07.052718    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:07.204185 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:07.204155    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:07.344755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:07.343552    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:07.554277 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:07.554237    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"]
Nov 03 10:48:07.565167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:07.565132    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-zv8d6"]
Nov 03 10:48:07.773325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:07.773285    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:07.902468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:07.902434    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a01722-f51b-4932-9f6f-81511daefeb4" path="/var/lib/kubelet/pods/12a01722-f51b-4932-9f6f-81511daefeb4/volumes"
Nov 03 10:48:08.553525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:08.549191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:09.157836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:09.157802    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:09.800961 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:09.800928    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:10.317551 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:10.317512    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:10.474273 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:10.474241    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:10.611862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:10.611774    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:10.763615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:10.763558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:11.032385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:11.032353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:11.633339 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:11.633304    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:11.719292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:11.719249    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"]
Nov 03 10:48:11.724438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:11.724411    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-273012-zqbcr"]
Nov 03 10:48:11.776310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:11.776247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:11.908250 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:11.908105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:11.969033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:11.969003    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643de369-2c0b-4ef9-ae0b-d87cda35ba3b" path="/var/lib/kubelet/pods/643de369-2c0b-4ef9-ae0b-d87cda35ba3b/volumes"
Nov 03 10:48:12.453435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:12.453396    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:12.941152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:12.941119    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:13.714317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:13.714285    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:14.383196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:14.383156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:15.096638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:15.096557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:15.775885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:15.775839    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:16.475721 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:16.475687    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:16.944099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:16.944068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:17.096134 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:17.096090    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:17.243329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:17.243237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:18.307505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:18.307474    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:18.442520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:18.442476    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:18.582465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:18.582278    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:18.755061 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:18.755021    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:18.908252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:18.908169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:19.052594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:19.050490    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:19.467276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:19.467239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:19.931601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:19.931555    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:20.186639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:20.186528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:20.421397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:20.421364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:20.571893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:20.571776    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:21.451796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:21.451761    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:22.140016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:22.139981    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:22.417387 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:48:22.417303    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:48:22.759973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:22.759933    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:23.523199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:23.523159    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:24.143714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:24.143664    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:24.283132 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:24.283082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:24.433758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:24.433722    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:24.571888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:24.571861    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:25.162734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:25.162686    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:25.638861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:25.638829    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:25.890350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:25.890022    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:26.155633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:26.155527    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:26.950447 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:26.950412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:28.836124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:28.836092    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:29.354594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:29.354548    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:29.971001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:29.970969    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:30.114725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:30.113163    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:30.420062 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:30.419973    5817 scope.go:122] "RemoveContainer" containerID="b70d6b7c46005325c43962c7618dde5abbd2f297cd36ebab1b2ff390d91fe2a7"
Nov 03 10:48:30.431830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:30.431809    5817 scope.go:122] "RemoveContainer" containerID="d62ec2b2c39aad0c596d3cbe190be125ce86a87e6d1e60af7d54435ade03fd07"
Nov 03 10:48:30.441633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:30.441611    5817 scope.go:122] "RemoveContainer" containerID="c04e832fe02ec68555715fc6407784e58c1de170968e12e86a6df05b5768e7cd"
Nov 03 10:48:30.452477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:30.452453    5817 scope.go:122] "RemoveContainer" containerID="618f736258153a0b23e52fca972c2a2fda4297574b157669fc78c9337667b5e9"
Nov 03 10:48:30.928875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:30.928843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:31.648641 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:31.648611    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:32.611318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:32.611280    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:33.460846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:33.459018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:33.594791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:33.594758    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:33.881138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:33.881048    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:34.204335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:34.203066    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:34.669752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:34.669640    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:35.259903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:35.259462    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:35.994008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:35.993971    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:36.648974 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:36.648937    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:36.679962 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:36.679935    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:48:37.882071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:37.882037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:38.033222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:38.033190    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:38.267213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:38.267184    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:38.543238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:38.543154    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:39.288601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:39.284661    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded"
Nov 03 10:48:39.476930 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:39.470449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:39.684644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:39.669286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:40.054655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:40.049184    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:41.348600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:41.347719    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:41.552703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:41.552658    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:48:41.557856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:41.555509    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:48:41.578720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:41.578007    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:48:41.648648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:41.646791    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:48:41.708597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:41.704634    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:48:41.735591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:41.729255    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:48:41.854672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:41.854641    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:42.344540 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:42.344508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:43.146562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.146526    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:43.717667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.717637    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"]
Nov 03 10:48:43.718259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718246    5817 cpu_manager.go:414] "containerMap: removing container" podUID="427a43ca-e3f1-4616-be50-352afca82a2f" containerName="revoke-keys"
Nov 03 10:48:43.718304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718261    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427a43ca-e3f1-4616-be50-352afca82a2f" containerName="revoke-keys"
Nov 03 10:48:43.718346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718307    5817 cpu_manager.go:414] "containerMap: removing container" podUID="643de369-2c0b-4ef9-ae0b-d87cda35ba3b" containerName="revoke-keys"
Nov 03 10:48:43.718346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718312    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643de369-2c0b-4ef9-ae0b-d87cda35ba3b" containerName="revoke-keys"
Nov 03 10:48:43.718346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718328    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy"
Nov 03 10:48:43.718346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718333    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy"
Nov 03 10:48:43.718346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718338    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy"
Nov 03 10:48:43.718346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718342    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy"
Nov 03 10:48:43.718528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718420    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy"
Nov 03 10:48:43.718528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718427    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe18144-9897-4034-99ff-f5ddec642f0d" containerName="istio-proxy"
Nov 03 10:48:43.718528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718434    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="427a43ca-e3f1-4616-be50-352afca82a2f" containerName="revoke-keys"
Nov 03 10:48:43.718528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.718447    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="643de369-2c0b-4ef9-ae0b-d87cda35ba3b" containerName="revoke-keys"
Nov 03 10:48:43.727164 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.727132    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.731717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.731687    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"]
Nov 03 10:48:43.731856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.731811    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:48:43.731973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.731956    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:48:43.828601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.828318    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:43.836612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.834930    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.836612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.835060    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zht5\" (UniqueName: \"kubernetes.io/projected/9819f50d-feba-4226-a37c-b6a0a819f193-kube-api-access-8zht5\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.836612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.835351    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.936545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.936508    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.936789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.936654    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.936789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.936754    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zht5\" (UniqueName: \"kubernetes.io/projected/9819f50d-feba-4226-a37c-b6a0a819f193-kube-api-access-8zht5\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.937212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.937190    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.939199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.939176    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:43.944499 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:43.944480    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zht5\" (UniqueName: \"kubernetes.io/projected/9819f50d-feba-4226-a37c-b6a0a819f193-kube-api-access-8zht5\") pod \"maas-api-revoke-keys-e2e-mt-ab592834-w6wcm\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:44.081069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:44.081005    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:44.393549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:44.393525    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"]
Nov 03 10:48:44.394343 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:48:44.394316    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9819f50d_feba_4226_a37c_b6a0a819f193.slice/crio-6bfde15d973419077f32309bdca09e80e4ec72a577b5538632780d2bf0f395b3 WatchSource:0}: Error finding container 6bfde15d973419077f32309bdca09e80e4ec72a577b5538632780d2bf0f395b3: Status 404 returned error can't find the container with id 6bfde15d973419077f32309bdca09e80e4ec72a577b5538632780d2bf0f395b3
Nov 03 10:48:44.501023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:44.500984    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:44.825488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:44.825442    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:44.923729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:44.923692    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm" event={"ID":"9819f50d-feba-4226-a37c-b6a0a819f193","Type":"ContainerStarted","Data":"f5c3f2665269a9d856973a2593e248c386012077198f9dcc73f236d1930054e1"}
Nov 03 10:48:44.923922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:44.923752    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm" event={"ID":"9819f50d-feba-4226-a37c-b6a0a819f193","Type":"ContainerStarted","Data":"6bfde15d973419077f32309bdca09e80e4ec72a577b5538632780d2bf0f395b3"}
Nov 03 10:48:44.941193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:44.941146    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm" podStartSLOduration=1.941131964 podStartE2EDuration="1.941131964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:44.93949995 +0000 UTC m=+1155.145082012" watchObservedRunningTime="2026-11-03 10:48:44.941131964 +0000 UTC m=+1155.146714015"
Nov 03 10:48:45.132162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:45.132073    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:45.579249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:45.579213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:45.856335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:45.856243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:45.934543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:45.934513    5817 generic.go:356] "Generic (PLEG): container finished" podID="9819f50d-feba-4226-a37c-b6a0a819f193" containerID="f5c3f2665269a9d856973a2593e248c386012077198f9dcc73f236d1930054e1" exitCode=0
Nov 03 10:48:45.934781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:45.934649    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm" event={"ID":"9819f50d-feba-4226-a37c-b6a0a819f193","Type":"ContainerDied","Data":"f5c3f2665269a9d856973a2593e248c386012077198f9dcc73f236d1930054e1"}
Nov 03 10:48:46.002593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:46.001590    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:46.828753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:46.828720    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:46.967102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:46.967049    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:47.204325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.204301    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:47.310367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.310332    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9819f50d-feba-4226-a37c-b6a0a819f193-kube-api-access-8zht5\" (UniqueName: \"kubernetes.io/projected/9819f50d-feba-4226-a37c-b6a0a819f193-kube-api-access-8zht5\") pod \"9819f50d-feba-4226-a37c-b6a0a819f193\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") "
Nov 03 10:48:47.310541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.310385    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-openshift-ca\") pod \"9819f50d-feba-4226-a37c-b6a0a819f193\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") "
Nov 03 10:48:47.310541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.310476    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-xks-ca\") pod \"9819f50d-feba-4226-a37c-b6a0a819f193\" (UID: \"9819f50d-feba-4226-a37c-b6a0a819f193\") "
Nov 03 10:48:47.310874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.310843    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-openshift-ca" pod "9819f50d-feba-4226-a37c-b6a0a819f193" (UID: "9819f50d-feba-4226-a37c-b6a0a819f193"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:48:47.311348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.311326    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:47.312773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.312743    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-xks-ca" pod "9819f50d-feba-4226-a37c-b6a0a819f193" (UID: "9819f50d-feba-4226-a37c-b6a0a819f193"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:48:47.312913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.312849    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9819f50d-feba-4226-a37c-b6a0a819f193-kube-api-access-8zht5" pod "9819f50d-feba-4226-a37c-b6a0a819f193" (UID: "9819f50d-feba-4226-a37c-b6a0a819f193"). InnerVolumeSpecName "kube-api-access-8zht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:47.413230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.413144    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zht5\" (UniqueName: \"kubernetes.io/projected/9819f50d-feba-4226-a37c-b6a0a819f193-kube-api-access-8zht5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:47.413230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.413178    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9819f50d-feba-4226-a37c-b6a0a819f193-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:47.851724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.851685    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:47.975040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.975011    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"
Nov 03 10:48:47.975040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.975026    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm" event={"ID":"9819f50d-feba-4226-a37c-b6a0a819f193","Type":"ContainerDied","Data":"6bfde15d973419077f32309bdca09e80e4ec72a577b5538632780d2bf0f395b3"}
Nov 03 10:48:47.975259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:47.975075    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bfde15d973419077f32309bdca09e80e4ec72a577b5538632780d2bf0f395b3"
Nov 03 10:48:48.780771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:48.780716    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:49.204775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:49.204742    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:49.340470 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:49.340445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:49.527714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:49.523237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:50.327486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:50.327268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:50.815992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:50.815952    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:50.982976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:50.982948    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:51.247918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:51.247880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:51.909920 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:51.909886    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:52.446219 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:48:52.446182    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:48:52.507394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:52.507364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:53.138159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:53.138127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:53.634769 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:53.634742    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:53.811598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:53.805521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:53.969006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:53.968971    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:54.231314 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:54.231223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:54.381113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:54.381081    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:54.562342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:54.562245    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:55.726724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:55.726676    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:56.766669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:56.766634    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:56.901510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:56.901476    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:57.804832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:57.804791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:57.975207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:57.975170    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:58.112552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:58.112450    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:58.252770 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:58.252684    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:58.791965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:58.791930    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:59.038629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:59.035353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:59.168376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:59.168286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:59.287453 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:59.287411    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"]
Nov 03 10:48:59.292470 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:59.292440    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab592834-w6wcm"]
Nov 03 10:48:59.688346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:59.688301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:59.824608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:59.824557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:48:59.904975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:48:59.904940    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9819f50d-feba-4226-a37c-b6a0a819f193" path="/var/lib/kubelet/pods/9819f50d-feba-4226-a37c-b6a0a819f193/volumes"
Nov 03 10:49:00.537660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:00.537560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:00.674341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:00.674300    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:00.804872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:00.804788    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:00.952517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:00.952485    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"]
Nov 03 10:49:00.952775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:00.952750    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy" containerID="cri-o://9e457413c4f48ea3773b0713c065885b9e358955dcc7663d7d8ee2d40a563aaa" gracePeriod=30
Nov 03 10:49:01.039068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:01.039037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:01.587192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:01.587106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:02.036656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.036622    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:02.367710 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.367623    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"]
Nov 03 10:49:02.368611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.368585    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9819f50d-feba-4226-a37c-b6a0a819f193" containerName="revoke-keys"
Nov 03 10:49:02.368611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.368610    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9819f50d-feba-4226-a37c-b6a0a819f193" containerName="revoke-keys"
Nov 03 10:49:02.368825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.368765    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9819f50d-feba-4226-a37c-b6a0a819f193" containerName="revoke-keys"
Nov 03 10:49:02.372716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.372697    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.380370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.380349    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-de30a0-openshift-default-dockercfg-zrzjx\""
Nov 03 10:49:02.388270 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.388033    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"]
Nov 03 10:49:02.455661 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.455627    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:02.549595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.549505    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-credential-socket\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.549813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.549610    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-certs\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.549813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.549653    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a28440-6ec6-432e-9154-a0ad0692c771-istiod-ca-cert\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.549813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.549681    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt5f\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-kube-api-access-5dt5f\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.549813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.549761    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-istio-token\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.550048 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.549839    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-socket\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.550048 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.549883    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a28440-6ec6-432e-9154-a0ad0692c771-istio-podinfo\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.550157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.550060    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-data\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.550157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.550115    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-envoy\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.655767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.655661    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-socket\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.655767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.655733    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a28440-6ec6-432e-9154-a0ad0692c771-istio-podinfo\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.656290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.655906    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-data\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.656290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.655959    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-envoy\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.656290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.655999    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-credential-socket\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.656290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.656069    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-certs\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.656290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.656106    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a28440-6ec6-432e-9154-a0ad0692c771-istiod-ca-cert\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.656290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.656134    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt5f\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-kube-api-access-5dt5f\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.656290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.656214    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-istio-token\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.657019 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.656994    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-socket\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.665857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.660657    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-data\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.667798 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.667776    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-envoy\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.689642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.675759    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-credential-socket\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.689642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.677697    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a28440-6ec6-432e-9154-a0ad0692c771-istiod-ca-cert\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.689642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.678711    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-certs\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.689642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.686206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:02.689642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.686986    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a28440-6ec6-432e-9154-a0ad0692c771-istio-podinfo\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.728597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.709429    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-istio-token\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.728597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.712267    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt5f\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-kube-api-access-5dt5f\") pod \"e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") " pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:02.999078 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:02.999043    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:03.298617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:03.296992    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"]
Nov 03 10:49:03.299209 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:49:03.299181    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a28440_6ec6_432e_9154_a0ad0692c771.slice/crio-395a249f3f3cf67f6c15f0d5123ec77bba20c35a9b12dc232a8161db0922506d WatchSource:0}: Error finding container 395a249f3f3cf67f6c15f0d5123ec77bba20c35a9b12dc232a8161db0922506d: Status 404 returned error can't find the container with id 395a249f3f3cf67f6c15f0d5123ec77bba20c35a9b12dc232a8161db0922506d
Nov 03 10:49:03.303834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:03.303801    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:03.303931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:03.303868    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:03.303931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:03.303897    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:03.342369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:03.340983    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:03.791596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:03.791445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:04.176845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:04.176715    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" event={"ID":"00a28440-6ec6-432e-9154-a0ad0692c771","Type":"ContainerStarted","Data":"7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc"}
Nov 03 10:49:04.176845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:04.176786    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" event={"ID":"00a28440-6ec6-432e-9154-a0ad0692c771","Type":"ContainerStarted","Data":"395a249f3f3cf67f6c15f0d5123ec77bba20c35a9b12dc232a8161db0922506d"}
Nov 03 10:49:04.212089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:04.212028    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" podStartSLOduration=2.211999429 podStartE2EDuration="2.211999429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:04.211649207 +0000 UTC m=+1174.417231258" watchObservedRunningTime="2026-11-03 10:49:04.211999429 +0000 UTC m=+1174.417581481"
Nov 03 10:49:04.255223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:04.255187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:04.395602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:04.395551    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:04.557603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:04.555531    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:04.880413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:04.880247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:05.000276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.000237    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:05.007825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.007798    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:05.027625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.027556    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"]
Nov 03 10:49:05.034421 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.034389    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a3be1f5-xnzbw"]
Nov 03 10:49:05.097723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.097693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:05.188612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.188419    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:05.190549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.190525    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:05.232222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.232193    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:05.855230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.855194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:05.906592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:05.906546    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0361d514-284d-4774-bad5-65c9ce296669" path="/var/lib/kubelet/pods/0361d514-284d-4774-bad5-65c9ce296669/volumes"
Nov 03 10:49:06.228742 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.228709    5817 generic.go:356] "Generic (PLEG): container finished" podID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerID="9e457413c4f48ea3773b0713c065885b9e358955dcc7663d7d8ee2d40a563aaa" exitCode=0
Nov 03 10:49:06.229549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.229522    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" event={"ID":"0d39912f-e2d5-4961-a3f4-42d08009885a","Type":"ContainerDied","Data":"9e457413c4f48ea3773b0713c065885b9e358955dcc7663d7d8ee2d40a563aaa"}
Nov 03 10:49:06.229814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.229793    5817 scope.go:122] "RemoveContainer" containerID="2ae754ea1635d766e917c64d1f1daf6188d5535d8d242441ab9348e41627cd34"
Nov 03 10:49:06.259298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.259280    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:49:06.405665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-certs\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.405665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405636    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-podinfo\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.405914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405681    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-data\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.405914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405726    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-envoy\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.405914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405793    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-certs" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:06.405914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405812    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-token\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-token\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.406126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405928    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-kube-api-access-tx9nf\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-kube-api-access-tx9nf\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.406126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405962    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d39912f-e2d5-4961-a3f4-42d08009885a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39912f-e2d5-4961-a3f4-42d08009885a-istiod-ca-cert\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.406126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.405996    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-data" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:06.406126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.406047    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-socket\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.406365 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.406140    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-credential-socket\") pod \"0d39912f-e2d5-4961-a3f4-42d08009885a\" (UID: \"0d39912f-e2d5-4961-a3f4-42d08009885a\") "
Nov 03 10:49:06.406656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.406297    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d39912f-e2d5-4961-a3f4-42d08009885a-istiod-ca-cert" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:06.406656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.406522    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-socket" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:06.407030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.407008    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39912f-e2d5-4961-a3f4-42d08009885a-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.407122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.407035    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.407122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.407050    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.407122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.407064    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.407303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.407281    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-credential-socket" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:06.408585 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.408538    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-token" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:06.408692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.408561    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-envoy" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:06.409239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.409212    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-podinfo" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:49:06.409489 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.409472    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-kube-api-access-tx9nf" pod "0d39912f-e2d5-4961-a3f4-42d08009885a" (UID: "0d39912f-e2d5-4961-a3f4-42d08009885a"). InnerVolumeSpecName "kube-api-access-tx9nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:06.508935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.508900    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.508935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.508933    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.508935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.508942    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.509263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.509019    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx9nf\" (UniqueName: \"kubernetes.io/projected/0d39912f-e2d5-4961-a3f4-42d08009885a-kube-api-access-tx9nf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.509263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.509031    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39912f-e2d5-4961-a3f4-42d08009885a-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:06.752278 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.752243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:06.912593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:06.900100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:07.239130 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:07.239100    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"
Nov 03 10:49:07.239361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:07.239333    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc" event={"ID":"0d39912f-e2d5-4961-a3f4-42d08009885a","Type":"ContainerDied","Data":"527fbf4f155a2c4b211142f39a2e07f5b83ffc9559b98a7de04ed33fb5f34884"}
Nov 03 10:49:07.239498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:07.239484    5817 scope.go:122] "RemoveContainer" containerID="9e457413c4f48ea3773b0713c065885b9e358955dcc7663d7d8ee2d40a563aaa"
Nov 03 10:49:07.279703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:07.279675    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"]
Nov 03 10:49:07.288327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:07.288302    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-642fe5-openshift-default-76fb996475-967bc"]
Nov 03 10:49:07.624712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:07.623699    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:07.911979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:07.911883    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" path="/var/lib/kubelet/pods/0d39912f-e2d5-4961-a3f4-42d08009885a/volumes"
Nov 03 10:49:07.955538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:07.955502    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:08.095616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:08.091535    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:08.657270 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:08.656531    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:09.022330 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:09.022302    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"]
Nov 03 10:49:09.026556 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:09.026529    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b03480b-4ml2k"]
Nov 03 10:49:09.142968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:09.142928    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:09.274131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:09.274051    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:09.448327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:09.448291    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:09.917523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:09.917482    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fefcd06-9c73-4dcc-80eb-af699f0fb418" path="/var/lib/kubelet/pods/3fefcd06-9c73-4dcc-80eb-af699f0fb418/volumes"
Nov 03 10:49:10.250184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:10.250141    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:10.968049 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:10.968001    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:11.636558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:11.636523    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:12.367159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:12.367127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:13.145610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:13.145550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:13.282599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:13.279715    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:14.190657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:14.184788    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:14.822033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:14.821998    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:49:15.178277 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:15.178248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:17.426169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:17.426130    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:18.508890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:18.508854    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:18.652088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:18.652049    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:18.807657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:18.805118    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:19.592437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.592403    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"]
Nov 03 10:49:19.593350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.593329    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy"
Nov 03 10:49:19.593350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.593350    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy"
Nov 03 10:49:19.593539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.593452    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy"
Nov 03 10:49:19.596247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.596222    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.617726 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.617372    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:19.647254 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.647212    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"]
Nov 03 10:49:19.755783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.755748    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.755783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.755796    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k64dq\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-kube-api-access-k64dq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.756002 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.755923    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.756081 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.756064    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1a5397-8f6b-4415-a277-53804bb59b46-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.756176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.756158    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.756301 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.756281    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.756383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.756367    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.858083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.858011    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.858083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.858043    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k64dq\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-kube-api-access-k64dq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.858281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.858090    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.858281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.858256    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1a5397-8f6b-4415-a277-53804bb59b46-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.858390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.858324    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.858448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.858420    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.858548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.858484    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.859349 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.858770    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.860913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.860890    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.861038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.860968    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1a5397-8f6b-4415-a277-53804bb59b46-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.861106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.861073    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.861106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.861099    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.870091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.870063    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k64dq\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-kube-api-access-k64dq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.872321 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.872296    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8jg4\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.947616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.947564    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:19.989158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:19.989122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:20.037337 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.036719    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"]
Nov 03 10:49:20.040911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.040888    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd8b1cb-tvlzd"]
Nov 03 10:49:20.179707 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.171461    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:20.295502 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.282729    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"]
Nov 03 10:49:20.295502 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.290564    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:20.295502 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.290675    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:20.482308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.481316    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4" event={"ID":"6a1a5397-8f6b-4415-a277-53804bb59b46","Type":"ContainerStarted","Data":"1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048"}
Nov 03 10:49:20.482308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.481392    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4" event={"ID":"6a1a5397-8f6b-4415-a277-53804bb59b46","Type":"ContainerStarted","Data":"57712cc7d8a329a3637e569a0fdd1fb64dd27b4b5a29a5fc5269418a2c8ccafe"}
Nov 03 10:49:20.482308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.481661    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:20.525430 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.525380    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4" podStartSLOduration=1.525348904 podStartE2EDuration="1.525348904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:20.520224757 +0000 UTC m=+1190.725806808" watchObservedRunningTime="2026-11-03 10:49:20.525348904 +0000 UTC m=+1190.730930954"
Nov 03 10:49:20.886192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.886120    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"]
Nov 03 10:49:20.886929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.886908    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy"
Nov 03 10:49:20.886929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.886930    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy"
Nov 03 10:49:20.887114 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.887101    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d39912f-e2d5-4961-a3f4-42d08009885a" containerName="istio-proxy"
Nov 03 10:49:20.896957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.896928    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"]
Nov 03 10:49:20.899762 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.899628    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:20.907553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.906498    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:49:20.907553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.906839    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:49:20.979177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.979141    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:20.999034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.998911    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:20.999696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.999668    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:20.999921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:20.999900    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2v9d\" (UniqueName: \"kubernetes.io/projected/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-kube-api-access-j2v9d\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:21.111040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.108271    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:21.111040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.108400    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:21.111040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.108565    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2v9d\" (UniqueName: \"kubernetes.io/projected/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-kube-api-access-j2v9d\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:21.111040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.110203    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:21.121103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.121070    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:21.123764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.123732    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2v9d\" (UniqueName: \"kubernetes.io/projected/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-kube-api-access-j2v9d\") pod \"maas-api-revoke-keys-e2e-api-a-90bad4-c794b\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:21.218814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.218784    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:21.500861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.500790    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 10:49:21.519324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.519293    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"]
Nov 03 10:49:21.525209 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:49:21.525019    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07979a2_df77_4ae9_9c7d_7eee3b5e51a5.slice/crio-38f91b27dfdfd90749ee549f90a514958f09b5ed9f22daa4843b72bd2d965376 WatchSource:0}: Error finding container 38f91b27dfdfd90749ee549f90a514958f09b5ed9f22daa4843b72bd2d965376: Status 404 returned error can't find the container with id 38f91b27dfdfd90749ee549f90a514958f09b5ed9f22daa4843b72bd2d965376
Nov 03 10:49:21.905276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:21.905247    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a772b33f-fc26-4016-a727-07adf1fd3e5d" path="/var/lib/kubelet/pods/a772b33f-fc26-4016-a727-07adf1fd3e5d/volumes"
Nov 03 10:49:22.080066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:22.079933    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:22.413662 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:49:22.413584    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:49:22.509507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:22.509457    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b" event={"ID":"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5","Type":"ContainerStarted","Data":"14cbf2cbb95b5c0e255572eb7af8fce35e3d1528258ac7f58dc3222d0c4d4150"}
Nov 03 10:49:22.509709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:22.509527    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b" event={"ID":"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5","Type":"ContainerStarted","Data":"38f91b27dfdfd90749ee549f90a514958f09b5ed9f22daa4843b72bd2d965376"}
Nov 03 10:49:22.526631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:22.526556    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b" podStartSLOduration=2.526540432 podStartE2EDuration="2.526540432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:22.523849013 +0000 UTC m=+1192.729431067" watchObservedRunningTime="2026-11-03 10:49:22.526540432 +0000 UTC m=+1192.732122482"
Nov 03 10:49:22.895425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:22.895390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:23.519322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:23.519282    5817 generic.go:356] "Generic (PLEG): container finished" podID="c07979a2-df77-4ae9-9c7d-7eee3b5e51a5" containerID="14cbf2cbb95b5c0e255572eb7af8fce35e3d1528258ac7f58dc3222d0c4d4150" exitCode=0
Nov 03 10:49:23.519834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:23.519358    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b" event={"ID":"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5","Type":"ContainerDied","Data":"14cbf2cbb95b5c0e255572eb7af8fce35e3d1528258ac7f58dc3222d0c4d4150"}
Nov 03 10:49:23.543420 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:23.543387    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:24.183466 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.183432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:24.775112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.774766    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:24.907034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.906952    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-xks-ca\") pod \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") "
Nov 03 10:49:24.907679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.907657    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-kube-api-access-j2v9d\" (UniqueName: \"kubernetes.io/projected/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-kube-api-access-j2v9d\") pod \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") "
Nov 03 10:49:24.908088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.908070    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-openshift-ca\") pod \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\" (UID: \"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5\") "
Nov 03 10:49:24.917263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.908555    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-openshift-ca" pod "c07979a2-df77-4ae9-9c7d-7eee3b5e51a5" (UID: "c07979a2-df77-4ae9-9c7d-7eee3b5e51a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:24.925604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.922194    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-xks-ca" pod "c07979a2-df77-4ae9-9c7d-7eee3b5e51a5" (UID: "c07979a2-df77-4ae9-9c7d-7eee3b5e51a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:49:24.939719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.939321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:24.941623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:24.941536    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-kube-api-access-j2v9d" pod "c07979a2-df77-4ae9-9c7d-7eee3b5e51a5" (UID: "c07979a2-df77-4ae9-9c7d-7eee3b5e51a5"). InnerVolumeSpecName "kube-api-access-j2v9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:25.018984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.018952    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:25.018984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.018980    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:25.019236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.018996    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2v9d\" (UniqueName: \"kubernetes.io/projected/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5-kube-api-access-j2v9d\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:25.278836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.278754    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:25.444598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.434068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:25.598657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.598504    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:25.601624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.601559    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b" event={"ID":"c07979a2-df77-4ae9-9c7d-7eee3b5e51a5","Type":"ContainerDied","Data":"38f91b27dfdfd90749ee549f90a514958f09b5ed9f22daa4843b72bd2d965376"}
Nov 03 10:49:25.601624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.601626    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38f91b27dfdfd90749ee549f90a514958f09b5ed9f22daa4843b72bd2d965376"
Nov 03 10:49:25.601849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:25.601697    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"
Nov 03 10:49:26.699068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:26.699033    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:27.886136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:27.886100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:28.035580 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:28.035522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:28.174247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:28.174135    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:28.970961 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:28.970909    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:29.102946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:29.102911    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:29.444999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:29.444966    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:29.610148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:29.610116    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:30.334101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:30.334024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:30.606442 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:30.606356    5817 scope.go:122] "RemoveContainer" containerID="cfbe75e9f39ee53c30f252a9d4638e013c63692e8f60a8ec432aa8529011c43b"
Nov 03 10:49:30.647716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:30.647680    5817 scope.go:122] "RemoveContainer" containerID="03910994422c85882a9bc0f43922bbe70926a3d1c6bded0bf50c03d48d67869a"
Nov 03 10:49:30.663515 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:30.663477    5817 scope.go:122] "RemoveContainer" containerID="45621c29dc8ff9e4b3e0772046739235970cbc117e4ecf42ad2fa1fa2072d443"
Nov 03 10:49:30.741785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:30.734968    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:30.913652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:30.912626    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:31.180804 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:31.180774    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:31.314734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:31.309515    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:32.052475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:32.052439    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:32.882588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:32.878464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:33.625592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:33.622143    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:34.439453 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:34.437075    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:35.373336 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:35.372237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:36.176026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:36.175987    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:37.615592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:37.614486    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:38.138398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:38.138367    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:38.272144 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:38.272105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:38.873058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:38.872490    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:39.744496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:39.744469    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:39.877983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:39.877952    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:40.011198 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:40.011108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:40.712593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:40.710206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:41.235011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:41.234979    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:41.856352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:41.855085    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:42.017882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:42.009980    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:42.152596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:42.152471    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:42.557598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:42.554310    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:43.276026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:43.275309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:43.961632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:43.961602    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:44.776644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:44.771019    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:45.543523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:45.543485    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:46.356480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:46.356410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:46.918891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:46.918858    5817 generic.go:356] "Generic (PLEG): container finished" podID="00a28440-6ec6-432e-9154-a0ad0692c771" containerID="7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc" exitCode=0
Nov 03 10:49:46.919315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:46.918923    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" event={"ID":"00a28440-6ec6-432e-9154-a0ad0692c771","Type":"ContainerDied","Data":"7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc"}
Nov 03 10:49:46.919382 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:46.919333    5817 scope.go:122] "RemoveContainer" containerID="7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc"
Nov 03 10:49:46.921027 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:46.920999    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:46.921123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:46.921066    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:46.921123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:46.921099    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:47.235783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:47.235122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:47.874242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:47.874214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:47.933305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:47.933270    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" event={"ID":"00a28440-6ec6-432e-9154-a0ad0692c771","Type":"ContainerStarted","Data":"a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5"}
Nov 03 10:49:47.999252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:47.999224    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:48.018373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.018232    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:48.153718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.153628    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"]
Nov 03 10:49:48.171757 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.171683    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:48.197599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.193137    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90bad4-c794b"]
Nov 03 10:49:48.369795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.369763    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:48.509124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.509091    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:48.635668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.635638    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"]
Nov 03 10:49:48.635891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.635865    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy" containerID="cri-o://a328e82241cacc104649f4d44d20ea61014e90af74b49667f0db015b827140cf" gracePeriod=30
Nov 03 10:49:48.662999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.661903    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"]
Nov 03 10:49:48.662999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.662130    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy" containerID="cri-o://30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361" gracePeriod=30
Nov 03 10:49:48.666283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.666261    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"]
Nov 03 10:49:48.666860 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.666681    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy" containerID="cri-o://742de57737f35ef1919d46dea7ebc471d72d5ed981decfba1bec87d460df671f" gracePeriod=30
Nov 03 10:49:48.734552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.732705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:48.881851 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:48.881767    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:49.023593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:49.023422    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 10:49:49.023593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:49.023494    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 10:49:49.039630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:49.024859    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:49.202595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:49.199921    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:49.346122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:49.346084    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:49.569279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:49.569173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:49.924626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:49.924510    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07979a2-df77-4ae9-9c7d-7eee3b5e51a5" path="/var/lib/kubelet/pods/c07979a2-df77-4ae9-9c7d-7eee3b5e51a5/volumes"
Nov 03 10:49:50.002702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:50.002653    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 10:49:50.002966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:50.002940    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 10:49:50.195615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:50.194691    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:50.437638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:50.425588    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:50.618877 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:50.618785    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:50.756592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:50.756021    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:51.003986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:51.003953    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 10:49:51.004311 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:51.004271    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 10:49:51.029591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:51.029548    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"]
Nov 03 10:49:51.036907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:51.036883    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b799193b-nx97m"]
Nov 03 10:49:51.316653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:51.301706    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:51.451595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:51.447307    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:51.921806 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:51.921774    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3f1b62-c382-493f-a22a-22ad7f3a5584" path="/var/lib/kubelet/pods/7c3f1b62-c382-493f-a22a-22ad7f3a5584/volumes"
Nov 03 10:49:51.951544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:51.951456    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:52.004705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.004671    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 10:49:52.004969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.004941    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 10:49:52.023813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.023780    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"]
Nov 03 10:49:52.031124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.028834    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c07979a2-df77-4ae9-9c7d-7eee3b5e51a5" containerName="revoke-keys"
Nov 03 10:49:52.031350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.031328    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07979a2-df77-4ae9-9c7d-7eee3b5e51a5" containerName="revoke-keys"
Nov 03 10:49:52.031747 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.031728    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07979a2-df77-4ae9-9c7d-7eee3b5e51a5" containerName="revoke-keys"
Nov 03 10:49:52.049504 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.049467    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.054557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.054530    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"]
Nov 03 10:49:52.070031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.070011    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:49:52.070158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.070091    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:49:52.179967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.179930    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.180163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.180124    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.180361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.180339    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mwz\" (UniqueName: \"kubernetes.io/projected/f68457f8-5de8-4e6b-adfb-e4597ea247a7-kube-api-access-p7mwz\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.282385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.282350    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mwz\" (UniqueName: \"kubernetes.io/projected/f68457f8-5de8-4e6b-adfb-e4597ea247a7-kube-api-access-p7mwz\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.282613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.282539    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.282710 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.282658    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.283298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.283270    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.285095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.285072    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.326046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.326011    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mwz\" (UniqueName: \"kubernetes.io/projected/f68457f8-5de8-4e6b-adfb-e4597ea247a7-kube-api-access-p7mwz\") pod \"maas-api-revoke-sub-9781a50d-6skj6\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.376486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.376459    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:52.435090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.435058    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:49:52.435239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.435124    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:49:52.471794 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:49:52.471754    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:49:52.555448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.554861    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:49:52.555448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.554912    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:49:52.750831 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.746685    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:52.821627 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:49:52.817223    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68457f8_5de8_4e6b_adfb_e4597ea247a7.slice/crio-fdffdd1567c9f240f7e1a7bef6fe66a5f0fa9c5fc9cf93c5b3402d067cd63bcb WatchSource:0}: Error finding container fdffdd1567c9f240f7e1a7bef6fe66a5f0fa9c5fc9cf93c5b3402d067cd63bcb: Status 404 returned error can't find the container with id fdffdd1567c9f240f7e1a7bef6fe66a5f0fa9c5fc9cf93c5b3402d067cd63bcb
Nov 03 10:49:52.840263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.839308    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"]
Nov 03 10:49:52.993598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:52.993497    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6" event={"ID":"f68457f8-5de8-4e6b-adfb-e4597ea247a7","Type":"ContainerStarted","Data":"fdffdd1567c9f240f7e1a7bef6fe66a5f0fa9c5fc9cf93c5b3402d067cd63bcb"}
Nov 03 10:49:53.012176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:53.011664    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 10:49:53.012176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:53.011734    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 10:49:53.441647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:53.438846    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:53.698748 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:53.698716    5817 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:49:53.698869 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:53.698777    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 10:49:53.881401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:53.881289    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:53.965970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:53.965940    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:49:54.018186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.014770    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:54.027126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.020148    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:49:54.036093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.028380    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6" event={"ID":"f68457f8-5de8-4e6b-adfb-e4597ea247a7","Type":"ContainerStarted","Data":"68ab68ba2f9d3118ab75bae0611d2245fe0499ba2224ac58ea51b26d8ad98f70"}
Nov 03 10:49:54.036093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.031903    5817 generic.go:356] "Generic (PLEG): container finished" podID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerID="30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361" exitCode=0
Nov 03 10:49:54.036093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.032008    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" event={"ID":"d0086c91-894b-4b31-8eb7-7c147e4512ba","Type":"ContainerDied","Data":"30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361"}
Nov 03 10:49:54.036093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.032098    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb" event={"ID":"d0086c91-894b-4b31-8eb7-7c147e4512ba","Type":"ContainerDied","Data":"e5085f23e849be568e10e254e11619a3481e4b1a68e8c9e37d4721d45d07f6ad"}
Nov 03 10:49:54.036093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.032129    5817 scope.go:122] "RemoveContainer" containerID="30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361"
Nov 03 10:49:54.036093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.032278    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"
Nov 03 10:49:54.036093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.034690    5817 generic.go:356] "Generic (PLEG): container finished" podID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerID="742de57737f35ef1919d46dea7ebc471d72d5ed981decfba1bec87d460df671f" exitCode=0
Nov 03 10:49:54.036093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.034776    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" event={"ID":"efb29c7e-ad83-4646-b10f-39f54cba1460","Type":"ContainerDied","Data":"742de57737f35ef1919d46dea7ebc471d72d5ed981decfba1bec87d460df671f"}
Nov 03 10:49:54.042783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.042760    5817 generic.go:356] "Generic (PLEG): container finished" podID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerID="a328e82241cacc104649f4d44d20ea61014e90af74b49667f0db015b827140cf" exitCode=0
Nov 03 10:49:54.043661 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.043644    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" event={"ID":"a691d134-0c27-4517-ba67-d76ab40cd5a6","Type":"ContainerDied","Data":"a328e82241cacc104649f4d44d20ea61014e90af74b49667f0db015b827140cf"}
Nov 03 10:49:54.078283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.078211    5817 scope.go:122] "RemoveContainer" containerID="80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0"
Nov 03 10:49:54.088831 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.088405    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6" podStartSLOduration=2.088385947 podStartE2EDuration="2.088385947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:54.084741736 +0000 UTC m=+1224.290323788" watchObservedRunningTime="2026-11-03 10:49:54.088385947 +0000 UTC m=+1224.293967998"
Nov 03 10:49:54.101121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.100693    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:49:54.105000 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.104273    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:49:54.133958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.133787    5817 scope.go:122] "RemoveContainer" containerID="30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361"
Nov 03 10:49:54.134550 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:49:54.134525    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361\": container with ID starting with 30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361 not found: ID does not exist" containerID="30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361"
Nov 03 10:49:54.134710 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.134686    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361"} err="failed to get container status \"30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361\": rpc error: code = NotFound desc = could not find container \"30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361\": container with ID starting with 30cfb823c29af59d537ce28f1b9f37dc7917c71c91061924879d55d889d5c361 not found: ID does not exist"
Nov 03 10:49:54.134815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.134805    5817 scope.go:122] "RemoveContainer" containerID="80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0"
Nov 03 10:49:54.135245 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:49:54.135183    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0\": container with ID starting with 80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0 not found: ID does not exist" containerID="80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0"
Nov 03 10:49:54.135245 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.135211    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0"} err="failed to get container status \"80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0\": rpc error: code = NotFound desc = could not find container \"80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0\": container with ID starting with 80162ca4dcc1b8dd2f0e5a62db21e4f637984647633e0872c7585f4fe61fb0e0 not found: ID does not exist"
Nov 03 10:49:54.150031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.150002    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-token\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-token\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.150142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.150088    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-socket\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.150203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.150152    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-certs\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.150263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.150230    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-data\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.150316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.150289    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0086c91-894b-4b31-8eb7-7c147e4512ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0086c91-894b-4b31-8eb7-7c147e4512ba-istiod-ca-cert\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.150408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.150371    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-envoy\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.150472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.150448    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-credential-socket\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.150546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.150528    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-podinfo\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.154195 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.154164    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-data" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.154643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.154619    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-kube-api-access-4pbhw\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-kube-api-access-4pbhw\") pod \"d0086c91-894b-4b31-8eb7-7c147e4512ba\" (UID: \"d0086c91-894b-4b31-8eb7-7c147e4512ba\") "
Nov 03 10:49:54.157699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.156145    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.157699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.157066    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-socket" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.157699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.157100    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0086c91-894b-4b31-8eb7-7c147e4512ba-istiod-ca-cert" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:54.157699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.157281    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-certs" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.157966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.157785    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-credential-socket" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.158643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.158353    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-token" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.159967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.159937    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-envoy" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.161189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.161153    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-kube-api-access-4pbhw" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "kube-api-access-4pbhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.164390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.164364    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-podinfo" pod "d0086c91-894b-4b31-8eb7-7c147e4512ba" (UID: "d0086c91-894b-4b31-8eb7-7c147e4512ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:49:54.171255 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.171227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:54.257413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257383    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-podinfo\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.257639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257515    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-credential-socket\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.257639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257544    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-data\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.257639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257636    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-data\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.257830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257693    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-socket\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.257830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257755    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-envoy\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.257926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257848    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-kube-api-access-hh2nw\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-kube-api-access-hh2nw\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.257926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257902    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-certs\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.258028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.257939    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-credential-socket\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.258549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.258447    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-socket\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.258549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.258516    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-certs\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.258754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.258544    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-data" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.258916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.258897    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-certs" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.259023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.258991    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-credential-socket" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.259408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259254    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-socket" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.259408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259312    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-credential-socket" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.259555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259424    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-data" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.259555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259431    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-token\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-token\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.259701 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259566    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-kube-api-access-pxn6p\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-kube-api-access-pxn6p\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.259701 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259656    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-socket" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.259811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259705    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-token\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-token\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.259811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259778    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb29c7e-ad83-4646-b10f-39f54cba1460-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb29c7e-ad83-4646-b10f-39f54cba1460-istiod-ca-cert\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.259917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259874    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-envoy\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.259917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259911    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-podinfo\") pod \"efb29c7e-ad83-4646-b10f-39f54cba1460\" (UID: \"efb29c7e-ad83-4646-b10f-39f54cba1460\") "
Nov 03 10:49:54.260032 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.259952    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a691d134-0c27-4517-ba67-d76ab40cd5a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a691d134-0c27-4517-ba67-d76ab40cd5a6-istiod-ca-cert\") pod \"a691d134-0c27-4517-ba67-d76ab40cd5a6\" (UID: \"a691d134-0c27-4517-ba67-d76ab40cd5a6\") "
Nov 03 10:49:54.260364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.260128    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-certs" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.260917    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.260942    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pbhw\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-kube-api-access-4pbhw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.260958    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.260971    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.260983    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.260998    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261013    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261027    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261041    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261055    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0086c91-894b-4b31-8eb7-7c147e4512ba-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261073    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261086    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261086    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb29c7e-ad83-4646-b10f-39f54cba1460-istiod-ca-cert" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261100    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261115    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261128    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261130    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-envoy" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.261158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261142    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0086c91-894b-4b31-8eb7-7c147e4512ba-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.262247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.261458    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a691d134-0c27-4517-ba67-d76ab40cd5a6-istiod-ca-cert" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:54.262247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.262101    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-podinfo" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:49:54.264181 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.264155    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-kube-api-access-pxn6p" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "kube-api-access-pxn6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.264290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.264199    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-token" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.264290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.264216    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-kube-api-access-hh2nw" pod "a691d134-0c27-4517-ba67-d76ab40cd5a6" (UID: "a691d134-0c27-4517-ba67-d76ab40cd5a6"). InnerVolumeSpecName "kube-api-access-hh2nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.265146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.265118    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-token" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.265146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.265131    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-envoy" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.266153 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.266134    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-podinfo" pod "efb29c7e-ad83-4646-b10f-39f54cba1460" (UID: "efb29c7e-ad83-4646-b10f-39f54cba1460"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:49:54.331583 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.331524    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:54.358046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.358017    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"]
Nov 03 10:49:54.362764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362742    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-mnhwb"]
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362789    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362805    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb29c7e-ad83-4646-b10f-39f54cba1460-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362815    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362824    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb29c7e-ad83-4646-b10f-39f54cba1460-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362833    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a691d134-0c27-4517-ba67-d76ab40cd5a6-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362842    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362851    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362861    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh2nw\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-kube-api-access-hh2nw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362869    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a691d134-0c27-4517-ba67-d76ab40cd5a6-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.362873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.362878    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxn6p\" (UniqueName: \"kubernetes.io/projected/efb29c7e-ad83-4646-b10f-39f54cba1460-kube-api-access-pxn6p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.486298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.486117    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:54.742630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.739382    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:54.872323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:54.872288    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:55.020657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.013653    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:55.091148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.091111    5817 generic.go:356] "Generic (PLEG): container finished" podID="f68457f8-5de8-4e6b-adfb-e4597ea247a7" containerID="68ab68ba2f9d3118ab75bae0611d2245fe0499ba2224ac58ea51b26d8ad98f70" exitCode=0
Nov 03 10:49:55.092156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.092128    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6" event={"ID":"f68457f8-5de8-4e6b-adfb-e4597ea247a7","Type":"ContainerDied","Data":"68ab68ba2f9d3118ab75bae0611d2245fe0499ba2224ac58ea51b26d8ad98f70"}
Nov 03 10:49:55.103399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.102870    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88" event={"ID":"efb29c7e-ad83-4646-b10f-39f54cba1460","Type":"ContainerDied","Data":"75d23bd68f66a6baef9aebee8f43611b415e0cb63dd903bf091d6378650bb604"}
Nov 03 10:49:55.103399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.102932    5817 scope.go:122] "RemoveContainer" containerID="742de57737f35ef1919d46dea7ebc471d72d5ed981decfba1bec87d460df671f"
Nov 03 10:49:55.103399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.103047    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"
Nov 03 10:49:55.118024 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.117137    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh" event={"ID":"a691d134-0c27-4517-ba67-d76ab40cd5a6","Type":"ContainerDied","Data":"9a45c650eb8d3fb8108659e90af8b1afde072a9d34cd33cb93dfa11f2606c8d4"}
Nov 03 10:49:55.118024 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.117550    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"
Nov 03 10:49:55.139270 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.139247    5817 scope.go:122] "RemoveContainer" containerID="a328e82241cacc104649f4d44d20ea61014e90af74b49667f0db015b827140cf"
Nov 03 10:49:55.145055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.145038    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"]
Nov 03 10:49:55.156932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.156901    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-8pk88"]
Nov 03 10:49:55.178703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.178609    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"]
Nov 03 10:49:55.181889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.181865    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-lrvgh"]
Nov 03 10:49:55.200771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.200732    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:55.327711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.327616    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:55.556528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.556492    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:55.699439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.699408    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:55.906898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.906495    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" path="/var/lib/kubelet/pods/a691d134-0c27-4517-ba67-d76ab40cd5a6/volumes"
Nov 03 10:49:55.907405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.907376    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" path="/var/lib/kubelet/pods/d0086c91-894b-4b31-8eb7-7c147e4512ba/volumes"
Nov 03 10:49:55.907937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:55.907922    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" path="/var/lib/kubelet/pods/efb29c7e-ad83-4646-b10f-39f54cba1460/volumes"
Nov 03 10:49:56.037000 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.036923    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"]
Nov 03 10:49:56.041392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.041359    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2f8cca-8bqjg"]
Nov 03 10:49:56.350596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.350558    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:56.418779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.418741    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68457f8-5de8-4e6b-adfb-e4597ea247a7-kube-api-access-p7mwz\" (UniqueName: \"kubernetes.io/projected/f68457f8-5de8-4e6b-adfb-e4597ea247a7-kube-api-access-p7mwz\") pod \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") "
Nov 03 10:49:56.418977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.418868    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-xks-ca\") pod \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") "
Nov 03 10:49:56.419251 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.419223    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-openshift-ca\") pod \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\" (UID: \"f68457f8-5de8-4e6b-adfb-e4597ea247a7\") "
Nov 03 10:49:56.419814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.419789    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-openshift-ca" pod "f68457f8-5de8-4e6b-adfb-e4597ea247a7" (UID: "f68457f8-5de8-4e6b-adfb-e4597ea247a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:56.420156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.420135    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:56.421211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.421184    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68457f8-5de8-4e6b-adfb-e4597ea247a7-kube-api-access-p7mwz" pod "f68457f8-5de8-4e6b-adfb-e4597ea247a7" (UID: "f68457f8-5de8-4e6b-adfb-e4597ea247a7"). InnerVolumeSpecName "kube-api-access-p7mwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:56.421331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.421308    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-xks-ca" pod "f68457f8-5de8-4e6b-adfb-e4597ea247a7" (UID: "f68457f8-5de8-4e6b-adfb-e4597ea247a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:49:56.522232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.522204    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7mwz\" (UniqueName: \"kubernetes.io/projected/f68457f8-5de8-4e6b-adfb-e4597ea247a7-kube-api-access-p7mwz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:56.522232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.522236    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68457f8-5de8-4e6b-adfb-e4597ea247a7-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:56.864209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:56.864176    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:57.003127 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:57.003094    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:57.141732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:57.141658    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"
Nov 03 10:49:57.141732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:57.141702    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6" event={"ID":"f68457f8-5de8-4e6b-adfb-e4597ea247a7","Type":"ContainerDied","Data":"fdffdd1567c9f240f7e1a7bef6fe66a5f0fa9c5fc9cf93c5b3402d067cd63bcb"}
Nov 03 10:49:57.141933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:57.141750    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdffdd1567c9f240f7e1a7bef6fe66a5f0fa9c5fc9cf93c5b3402d067cd63bcb"
Nov 03 10:49:57.860272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:57.860234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:57.922502 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:57.922468    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08feb4f-b2e1-4f4e-b23d-3859a661bdd3" path="/var/lib/kubelet/pods/b08feb4f-b2e1-4f4e-b23d-3859a661bdd3/volumes"
Nov 03 10:49:58.338348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:58.338308    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:58.497353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:58.497321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:58.777564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:58.777478    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:59.533361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:59.533326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:59.749701 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:59.749496    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:49:59.886499 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:49:59.886405    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:00.159238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:00.159147    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:50:00.947141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:00.947111    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:01.906419 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:01.906386    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:02.053710 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:02.053501    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:02.500600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:02.498714    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:03.135041 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:03.134975    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:04.029599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.018672    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7h7sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:50:04.029599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.018737    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.073695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.073658    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:50:04.073923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.073896    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.544253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.534448    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.688599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.686675    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.700597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.695664    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.754619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.749261    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:50:04.754619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.749268    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.754619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.749696    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.754619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.749296    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:50:04.772596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.765101    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.799648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.798742    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.883767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.883711    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:50:04.905694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:04.905650    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:06.213119 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:06.213082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:06.350681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:06.350637    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:06.481878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:06.481786    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:08.027929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:08.027893    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:08.755926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:08.755883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:09.019407 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:09.019324    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:09.883118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:09.883084    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:10.638756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:10.638722    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:11.339764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:11.339731    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:11.985618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:11.985555    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:12.114917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:12.114880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:12.301368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:12.301249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:12.963439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:12.963405    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:13.861645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:13.861607    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:14.659255 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:14.659222    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:15.668871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:15.668833    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:16.021078 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:16.021048    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"]
Nov 03 10:50:16.025103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:16.025070    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44884449-pbpb6"]
Nov 03 10:50:16.797292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:16.797260    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:17.686494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:17.686461    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:17.905614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:17.905535    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948d9d84-85d4-4a10-9873-674500d7ff77" path="/var/lib/kubelet/pods/948d9d84-85d4-4a10-9873-674500d7ff77/volumes"
Nov 03 10:50:18.585093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:18.585056    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:19.463272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:19.463234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:20.463954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:20.463899    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:20.615901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:20.615866    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:20.921529 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:20.921447    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:21.076620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:21.075783    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:21.962066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:21.962031    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:22.439540 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:50:22.439511    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:50:22.783982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:22.783901    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:23.822016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:23.821135    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:23.979012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:23.978981    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:50:24.806352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:24.806318    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:24.953209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:24.953159    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:25.092985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:25.092897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:25.459215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:25.459189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:26.041413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:26.041015    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:26.285749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:26.268539    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:27.065599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:27.054993    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"]
Nov 03 10:50:27.079262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:27.079235    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2396fbd2-pfmns"]
Nov 03 10:50:27.240647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:27.229550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:27.941932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:27.941896    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a2c7c0-c05f-4a5b-bc6b-b8c81a302422" path="/var/lib/kubelet/pods/38a2c7c0-c05f-4a5b-bc6b-b8c81a302422/volumes"
Nov 03 10:50:28.180713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:28.180650    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 03 10:50:28.535711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:28.535669    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:28.557700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:28.557664    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:28.558237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:28.558208    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": context deadline exceeded"
Nov 03 10:50:28.704871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:28.704821    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:28.739803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:28.739747    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:28.740520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:28.740490    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:50:28.930389 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:28.930363    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:29.182160 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:29.182078    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:29.335965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:29.335924    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:30.473507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:30.473467    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:30.749325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:30.749245    5817 scope.go:122] "RemoveContainer" containerID="257e5dc92d1d86dd6b04ce82f61b11a14189df7956cb79183c0a50d06e564fd2"
Nov 03 10:50:30.761004 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:30.760943    5817 scope.go:122] "RemoveContainer" containerID="b958985731dd364fb29153b73715383469c6f493b6174dd94071e5bb9226f484"
Nov 03 10:50:30.771383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:30.771363    5817 scope.go:122] "RemoveContainer" containerID="9a2fdca66fdd68f97f10e7f10f19707c07eb78b4e9e1c2b148f74312408f6316"
Nov 03 10:50:31.239488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:31.239454    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:32.087677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:32.087562    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:32.754310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:32.754274    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:32.904376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:32.904340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:33.046696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:33.044094    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:33.295873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:33.295764    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:33.502972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:33.502940    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:34.151357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:34.151323    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:35.046681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:35.046646    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:35.756362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:35.756321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:36.207871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:36.207843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:36.367022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:36.366990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:36.889304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:36.889275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:37.015625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:37.015588    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:38.027399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:38.027364    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"]
Nov 03 10:50:38.038549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:38.038520    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa4a73b-qgwkv"]
Nov 03 10:50:38.091464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:38.091431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:39.469413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:39.468597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:39.859687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:39.859544    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:39.953234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:39.953201    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b2ec74-dfdb-469e-9748-a8c0c43b1362" path="/var/lib/kubelet/pods/f7b2ec74-dfdb-469e-9748-a8c0c43b1362/volumes"
Nov 03 10:50:40.017598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:40.016137    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:40.313478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:40.313434    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:40.873362 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:40.873330    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:41.687222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:41.687154    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:41.884460 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:41.884417    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:42.140345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:42.140245    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:42.590648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:42.590541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:43.269401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:43.269355    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:43.987150 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:43.986390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:44.683629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:44.679946    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:45.349688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:45.349651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:46.085119 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:46.085084    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:46.643059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:46.643016    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:47.454989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:47.454953    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:48.426684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:48.426647    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:49.143965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.143933    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"]
Nov 03 10:50:49.144541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144526    5817 cpu_manager.go:414] "containerMap: removing container" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy"
Nov 03 10:50:49.144605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144541    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy"
Nov 03 10:50:49.144605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144583    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy"
Nov 03 10:50:49.144605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144591    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy"
Nov 03 10:50:49.144605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144604    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f68457f8-5de8-4e6b-adfb-e4597ea247a7" containerName="revoke-keys"
Nov 03 10:50:49.144736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144609    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68457f8-5de8-4e6b-adfb-e4597ea247a7" containerName="revoke-keys"
Nov 03 10:50:49.144736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144627    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy"
Nov 03 10:50:49.144736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144631    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy"
Nov 03 10:50:49.144837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144740    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy"
Nov 03 10:50:49.144837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144749    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68457f8-5de8-4e6b-adfb-e4597ea247a7" containerName="revoke-keys"
Nov 03 10:50:49.144837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144759    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a691d134-0c27-4517-ba67-d76ab40cd5a6" containerName="istio-proxy"
Nov 03 10:50:49.144837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.144768    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb29c7e-ad83-4646-b10f-39f54cba1460" containerName="istio-proxy"
Nov 03 10:50:49.148265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.148245    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.159839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.159818    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"]
Nov 03 10:50:49.160126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.160109    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a51d55-openshift-default-dockercfg-rpb7t\""
Nov 03 10:50:49.276698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.276665    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:49.279054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279022    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-credential-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.279194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279068    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-podinfo\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.279194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279099    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96596406-3fe4-45fb-add8-18425ca8f1e7-istiod-ca-cert\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.279291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279251    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-envoy\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.279343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279313    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-token\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.279400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279384    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-certs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.279541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279528    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.279612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279563    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6qs\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-kube-api-access-vq6qs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.279702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.279683    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-data\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.380795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.380765    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-token\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.380795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.380799    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-certs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.380887    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.380925    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6qs\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-kube-api-access-vq6qs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.380995    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-data\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381048    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-credential-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381079    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-podinfo\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381115    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96596406-3fe4-45fb-add8-18425ca8f1e7-istiod-ca-cert\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381177    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-envoy\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381414 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381301    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-certs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381414 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381356    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381413    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-credential-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381441    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-data\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.381908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.381858    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96596406-3fe4-45fb-add8-18425ca8f1e7-istiod-ca-cert\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.383733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.383707    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-envoy\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.384236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.384216    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-podinfo\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.389486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.389460    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-token\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.389618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.389498    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6qs\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-kube-api-access-vq6qs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.477278 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.477248    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:49.510604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.510537    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:49.687596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.679982    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:49.846861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.846774    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:49.914157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.914113    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:49.914298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.914203    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:49.914298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.914251    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:49.933190 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.933168    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"]
Nov 03 10:50:49.957745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:49.957716    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" event={"ID":"96596406-3fe4-45fb-add8-18425ca8f1e7","Type":"ContainerStarted","Data":"db1be2ca36343c5c747cd6605a3fece52e6481b8639fdca27fc2f6c532e02006"}
Nov 03 10:50:50.098542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:50.098470    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:50:50.276283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:50.276249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:50.966853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:50.966816    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" event={"ID":"96596406-3fe4-45fb-add8-18425ca8f1e7","Type":"ContainerStarted","Data":"c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0"}
Nov 03 10:50:51.017977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.017873    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podStartSLOduration=2.017849931 podStartE2EDuration="2.017849931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:50.985198694 +0000 UTC m=+1281.190780755" watchObservedRunningTime="2026-11-03 10:50:51.017849931 +0000 UTC m=+1281.223431983"
Nov 03 10:50:51.027319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.027284    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"]
Nov 03 10:50:51.032816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.032792    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cb09d0-nbd9v"]
Nov 03 10:50:51.411715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.411619    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"]
Nov 03 10:50:51.412670 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.412646    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy"
Nov 03 10:50:51.412670 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.412670    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy"
Nov 03 10:50:51.412887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.412872    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0086c91-894b-4b31-8eb7-7c147e4512ba" containerName="istio-proxy"
Nov 03 10:50:51.427859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.427835    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"]
Nov 03 10:50:51.428055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.427970    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.431990 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.431943    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:50:51.432225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.432012    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:50:51.477562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.477535    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:51.479186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.479164    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 10:50:51.479271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.479216    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 10:50:51.537067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.519023    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzddg\" (UniqueName: \"kubernetes.io/projected/e49d8019-2d8a-4814-a37c-07501e7f7127-kube-api-access-pzddg\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.537067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.519276    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.537067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.519425    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.547376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.547200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:51.620988 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.620948    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.621626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.621162    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.621626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.621231    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzddg\" (UniqueName: \"kubernetes.io/projected/e49d8019-2d8a-4814-a37c-07501e7f7127-kube-api-access-pzddg\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.621859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.621835    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.626564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.626534    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.630437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.630415    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzddg\" (UniqueName: \"kubernetes.io/projected/e49d8019-2d8a-4814-a37c-07501e7f7127-kube-api-access-pzddg\") pod \"maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.747194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.747162    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:51.934145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:51.933881    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0983ec40-cd53-44fe-bfbf-7a42f00e52f6" path="/var/lib/kubelet/pods/0983ec40-cd53-44fe-bfbf-7a42f00e52f6/volumes"
Nov 03 10:50:52.069973 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:50:52.069943    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49d8019_2d8a_4814_a37c_07501e7f7127.slice/crio-98278203ce543f2dc9c98c95cc9df9b1cf59edd38e7d72adcde4c29aceefda4e WatchSource:0}: Error finding container 98278203ce543f2dc9c98c95cc9df9b1cf59edd38e7d72adcde4c29aceefda4e: Status 404 returned error can't find the container with id 98278203ce543f2dc9c98c95cc9df9b1cf59edd38e7d72adcde4c29aceefda4e
Nov 03 10:50:52.070416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:52.070394    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"]
Nov 03 10:50:52.125320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:52.125283    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:52.442965 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:50:52.442909    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:50:52.478585 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:52.478546    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 10:50:52.478731 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:52.478608    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 10:50:52.550326 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:52.550214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:52.753604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:52.752269    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:52.897600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:52.897215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:53.044658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:53.040939    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g" event={"ID":"e49d8019-2d8a-4814-a37c-07501e7f7127","Type":"ContainerStarted","Data":"c6ccfa073d02d5b9f6e506307c4d030101e045e5b77df6419179c680e56f525d"}
Nov 03 10:50:53.044658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:53.041025    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g" event={"ID":"e49d8019-2d8a-4814-a37c-07501e7f7127","Type":"ContainerStarted","Data":"98278203ce543f2dc9c98c95cc9df9b1cf59edd38e7d72adcde4c29aceefda4e"}
Nov 03 10:50:53.083595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:53.082623    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g" podStartSLOduration=2.082603004 podStartE2EDuration="2.082603004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:53.070128845 +0000 UTC m=+1283.275710896" watchObservedRunningTime="2026-11-03 10:50:53.082603004 +0000 UTC m=+1283.288185056"
Nov 03 10:50:53.445251 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:53.445215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:53.478318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:53.478286    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 10:50:53.478526 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:53.478341    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 10:50:54.026986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:54.026955    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"]
Nov 03 10:50:54.034142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:54.034100    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f81728-m5pmh"]
Nov 03 10:50:54.054152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:54.054112    5817 generic.go:356] "Generic (PLEG): container finished" podID="e49d8019-2d8a-4814-a37c-07501e7f7127" containerID="c6ccfa073d02d5b9f6e506307c4d030101e045e5b77df6419179c680e56f525d" exitCode=0
Nov 03 10:50:54.054356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:54.054195    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g" event={"ID":"e49d8019-2d8a-4814-a37c-07501e7f7127","Type":"ContainerDied","Data":"c6ccfa073d02d5b9f6e506307c4d030101e045e5b77df6419179c680e56f525d"}
Nov 03 10:50:54.153964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:54.153929    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:54.477986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:54.477953    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 10:50:54.478366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:54.478002    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 10:50:54.865139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:54.864985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:55.329126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.329103    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:55.372541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.372513    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49d8019-2d8a-4814-a37c-07501e7f7127-kube-api-access-pzddg\" (UniqueName: \"kubernetes.io/projected/e49d8019-2d8a-4814-a37c-07501e7f7127-kube-api-access-pzddg\") pod \"e49d8019-2d8a-4814-a37c-07501e7f7127\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") "
Nov 03 10:50:55.372755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.372743    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-openshift-ca\") pod \"e49d8019-2d8a-4814-a37c-07501e7f7127\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") "
Nov 03 10:50:55.372805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.372780    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-xks-ca\") pod \"e49d8019-2d8a-4814-a37c-07501e7f7127\" (UID: \"e49d8019-2d8a-4814-a37c-07501e7f7127\") "
Nov 03 10:50:55.373107 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.373070    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-openshift-ca" pod "e49d8019-2d8a-4814-a37c-07501e7f7127" (UID: "e49d8019-2d8a-4814-a37c-07501e7f7127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:50:55.373256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.373204    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:55.375067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.375048    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-xks-ca" pod "e49d8019-2d8a-4814-a37c-07501e7f7127" (UID: "e49d8019-2d8a-4814-a37c-07501e7f7127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:50:55.375123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.375092    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49d8019-2d8a-4814-a37c-07501e7f7127-kube-api-access-pzddg" pod "e49d8019-2d8a-4814-a37c-07501e7f7127" (UID: "e49d8019-2d8a-4814-a37c-07501e7f7127"). InnerVolumeSpecName "kube-api-access-pzddg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:55.474486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.474446    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d8019-2d8a-4814-a37c-07501e7f7127-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:55.474486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.474484    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzddg\" (UniqueName: \"kubernetes.io/projected/e49d8019-2d8a-4814-a37c-07501e7f7127-kube-api-access-pzddg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:55.478034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.478015    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 10:50:55.478368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.478075    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 10:50:55.726169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.724156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:55.926810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:55.926767    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105d6801-0746-49de-a3c9-3a9252277f3f" path="/var/lib/kubelet/pods/105d6801-0746-49de-a3c9-3a9252277f3f/volumes"
Nov 03 10:50:56.075047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:56.074971    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g" event={"ID":"e49d8019-2d8a-4814-a37c-07501e7f7127","Type":"ContainerDied","Data":"98278203ce543f2dc9c98c95cc9df9b1cf59edd38e7d72adcde4c29aceefda4e"}
Nov 03 10:50:56.075047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:56.075019    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98278203ce543f2dc9c98c95cc9df9b1cf59edd38e7d72adcde4c29aceefda4e"
Nov 03 10:50:56.075224 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:56.075070    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"
Nov 03 10:50:56.364682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:56.364558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:56.482611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:56.478988    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 10:50:56.482611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:56.479045    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 10:50:56.515607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:56.513964    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:57.031899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:57.031866    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:57.268596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:57.267421    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:57.482904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:57.482876    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:57.483427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:57.483403    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:57.484478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:57.484456    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:50:57.617088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:57.617057    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:57.783713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:57.781882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:58.298352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:58.297704    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:58.874727 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:58.874693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:59.008279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:59.008243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:50:59.659557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:50:59.659521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:00.483904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:00.483486    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:01.300514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:01.300484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:02.096856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:02.096824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:02.927627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:02.925125    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:03.022471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:03.022442    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"]
Nov 03 10:51:03.026751 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:03.026723    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219808f0-29gmm"]
Nov 03 10:51:03.754987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:03.754951    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:03.903703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:03.903672    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f49dfd-7beb-4fdf-8c61-f61a06103f03" path="/var/lib/kubelet/pods/d8f49dfd-7beb-4fdf-8c61-f61a06103f03/volumes"
Nov 03 10:51:04.508832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:04.508784    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:05.135833 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:05.135799    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:05.764237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:05.764204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:05.889521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:05.889488    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:06.674919 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:06.674880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:07.413556 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:07.413521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:08.228300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:08.228252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:08.992978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:08.992918    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:09.693848 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:09.690096    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:10.384076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:10.384027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:10.537168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:10.537139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:10.637993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:10.637915    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"]
Nov 03 10:51:10.641405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:10.641383    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de30a0-89w5g"]
Nov 03 10:51:10.678349 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:10.678313    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:10.807960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:10.807925    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:11.015594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:11.011823    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:11.183438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:11.183389    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:11.902100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:11.902062    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49d8019-2d8a-4814-a37c-07501e7f7127" path="/var/lib/kubelet/pods/e49d8019-2d8a-4814-a37c-07501e7f7127/volumes"
Nov 03 10:51:12.157471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:12.157366    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:12.943072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:12.943041    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"]
Nov 03 10:51:12.943886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:12.943304    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy" containerID="cri-o://7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d" gracePeriod=30
Nov 03 10:51:12.949665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:12.949636    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"]
Nov 03 10:51:12.949881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:12.949858    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy" containerID="cri-o://fe98a4cd90dcfdcb7f742271acbb44ce3e67139f2de810783a1f660a51fb2e06" gracePeriod=30
Nov 03 10:51:13.088602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:13.082875    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:13.791390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:13.791354    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:14.485808 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:14.482800    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:15.268003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:15.267955    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:15.938183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:15.938151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:16.676118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:16.668925    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:16.999463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:16.999431    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"]
Nov 03 10:51:17.000254 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.000234    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e49d8019-2d8a-4814-a37c-07501e7f7127" containerName="revoke-keys"
Nov 03 10:51:17.000368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.000257    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49d8019-2d8a-4814-a37c-07501e7f7127" containerName="revoke-keys"
Nov 03 10:51:17.000484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.000470    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49d8019-2d8a-4814-a37c-07501e7f7127" containerName="revoke-keys"
Nov 03 10:51:17.004071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.004050    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.007234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.007211    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:51:17.008403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.008383    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:51:17.011919 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.010826    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"]
Nov 03 10:51:17.158155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.158124    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.158316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.158178    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggdm9\" (UniqueName: \"kubernetes.io/projected/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-kube-api-access-ggdm9\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.158316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.158288    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.259553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.259466    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.259553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.259531    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggdm9\" (UniqueName: \"kubernetes.io/projected/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-kube-api-access-ggdm9\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.259813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.259596    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.260351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.260281    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.262058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.262035    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.278625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.278598    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggdm9\" (UniqueName: \"kubernetes.io/projected/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-kube-api-access-ggdm9\") pod \"maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.325949 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.325919    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:17.529519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.529331    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:17.682597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.679209    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"]
Nov 03 10:51:17.696514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.696447    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:17.720850 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:51:17.720816    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f154d8_4d61_4d9b_bc60_92a3ef70fc03.slice/crio-ad8783911a441d790c3723a1ca53c9ffbc8c7026fdabe3b5cb487ab61df7bc3e WatchSource:0}: Error finding container ad8783911a441d790c3723a1ca53c9ffbc8c7026fdabe3b5cb487ab61df7bc3e: Status 404 returned error can't find the container with id ad8783911a441d790c3723a1ca53c9ffbc8c7026fdabe3b5cb487ab61df7bc3e
Nov 03 10:51:17.834748 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.834713    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"]
Nov 03 10:51:17.837281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.834962    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy" containerID="cri-o://a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5" gracePeriod=30
Nov 03 10:51:17.865122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.865074    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:17.998123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:17.998082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:18.352517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.352491    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"]
Nov 03 10:51:18.356543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.356522    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:51:18.357361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.357346    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.363753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.363727    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"]
Nov 03 10:51:18.415456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.415422    5817 generic.go:356] "Generic (PLEG): container finished" podID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerID="fe98a4cd90dcfdcb7f742271acbb44ce3e67139f2de810783a1f660a51fb2e06" exitCode=0
Nov 03 10:51:18.415754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.415727    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" event={"ID":"68b2d21f-effb-4929-8c92-c4d49af1499e","Type":"ContainerDied","Data":"fe98a4cd90dcfdcb7f742271acbb44ce3e67139f2de810783a1f660a51fb2e06"}
Nov 03 10:51:18.418660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.418636    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5" event={"ID":"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03","Type":"ContainerStarted","Data":"ad8783911a441d790c3723a1ca53c9ffbc8c7026fdabe3b5cb487ab61df7bc3e"}
Nov 03 10:51:18.420560 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.420536    5817 generic.go:356] "Generic (PLEG): container finished" podID="1541c102-d0ff-41af-805a-a15bc6951759" containerID="7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d" exitCode=0
Nov 03 10:51:18.420998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.420981    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" event={"ID":"1541c102-d0ff-41af-805a-a15bc6951759","Type":"ContainerDied","Data":"7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d"}
Nov 03 10:51:18.421126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.421112    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc" event={"ID":"1541c102-d0ff-41af-805a-a15bc6951759","Type":"ContainerDied","Data":"d80d6b6a41162005c16cf912539bd4c3dbbbdde89adfd76333336d435b1800f7"}
Nov 03 10:51:18.421235 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.421223    5817 scope.go:122] "RemoveContainer" containerID="7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d"
Nov 03 10:51:18.421479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.421466    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"
Nov 03 10:51:18.438299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.438249    5817 scope.go:122] "RemoveContainer" containerID="7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d"
Nov 03 10:51:18.438550 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:18.438528    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d\": container with ID starting with 7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d not found: ID does not exist" containerID="7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d"
Nov 03 10:51:18.438651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.438557    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d"} err="failed to get container status \"7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d\": rpc error: code = NotFound desc = could not find container \"7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d\": container with ID starting with 7a4748de2397dd24fb843943f41663b4e71e0c5158a498d33585760142d55c7d not found: ID does not exist"
Nov 03 10:51:18.444940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.444920    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-socket\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.444961    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1541c102-d0ff-41af-805a-a15bc6951759-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541c102-d0ff-41af-805a-a15bc6951759-istio-podinfo\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445022    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-certs\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445178    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-socket" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.445289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445276    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1541c102-d0ff-41af-805a-a15bc6951759-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541c102-d0ff-41af-805a-a15bc6951759-istiod-ca-cert\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445351    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-istio-token\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-istio-token\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445407 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445393    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-envoy\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445474    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-kube-api-access-pmlv8\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-kube-api-access-pmlv8\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445392    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-certs" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.445644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445564    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-credential-socket\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445600    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1541c102-d0ff-41af-805a-a15bc6951759-istiod-ca-cert" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:18.445644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445625    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-data\") pod \"1541c102-d0ff-41af-805a-a15bc6951759\" (UID: \"1541c102-d0ff-41af-805a-a15bc6951759\") "
Nov 03 10:51:18.445923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445899    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-credential-socket" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.446088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.445955    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.446303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.446262    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.446529 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.446509    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9rjd\" (UniqueName: \"kubernetes.io/projected/df10ca75-8590-4713-9b5e-f2642b7a7154-kube-api-access-z9rjd\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.446627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.446082    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-data" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.447005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.446982    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.447081 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.447011    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.447081 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.447028    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541c102-d0ff-41af-805a-a15bc6951759-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.447081 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.447042    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.447081 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.447056    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.447447 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.447421    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1541c102-d0ff-41af-805a-a15bc6951759-istio-podinfo" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:18.447705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.447687    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-envoy" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.447958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.447935    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-kube-api-access-pmlv8" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "kube-api-access-pmlv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:18.447958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.447947    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-istio-token" pod "1541c102-d0ff-41af-805a-a15bc6951759" (UID: "1541c102-d0ff-41af-805a-a15bc6951759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:18.548711 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.548670    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.548861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.548756    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.548861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.548822    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rjd\" (UniqueName: \"kubernetes.io/projected/df10ca75-8590-4713-9b5e-f2642b7a7154-kube-api-access-z9rjd\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.549190 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.549170    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.549265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.549197    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541c102-d0ff-41af-805a-a15bc6951759-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.549265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.549213    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmlv8\" (UniqueName: \"kubernetes.io/projected/1541c102-d0ff-41af-805a-a15bc6951759-kube-api-access-pmlv8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.549265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.549226    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541c102-d0ff-41af-805a-a15bc6951759-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.553426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.553385    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.558252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.558224    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.562120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.562100    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rjd\" (UniqueName: \"kubernetes.io/projected/df10ca75-8590-4713-9b5e-f2642b7a7154-kube-api-access-z9rjd\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.638055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.638033    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:51:18.682890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.682859    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.751790    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-socket\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.751852    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-kube-api-access-r2vn5\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-kube-api-access-r2vn5\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.751887    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b2d21f-effb-4929-8c92-c4d49af1499e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b2d21f-effb-4929-8c92-c4d49af1499e-istiod-ca-cert\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.752044    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-data\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.752109    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-credential-socket\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.752164    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-envoy\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.752233    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-certs\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.752338    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-token\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-token\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.752370    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-podinfo\") pod \"68b2d21f-effb-4929-8c92-c4d49af1499e\" (UID: \"68b2d21f-effb-4929-8c92-c4d49af1499e\") "
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.752651    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-data" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.753340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.752779    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-socket" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.754890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.754775    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-certs" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.754890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.754775    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b2d21f-effb-4929-8c92-c4d49af1499e-istiod-ca-cert" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:18.755293 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.755104    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-credential-socket" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.755645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.755566    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.755752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.755652    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.755752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.755671    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.755752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.755679    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.755752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.755690    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b2d21f-effb-4929-8c92-c4d49af1499e-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.763042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.763015    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-kube-api-access-r2vn5" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "kube-api-access-r2vn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:18.774336 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.774257    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-podinfo" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:18.774336 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.774319    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"]
Nov 03 10:51:18.775102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.775054    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-envoy" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:18.776625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.775560    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-token" pod "68b2d21f-effb-4929-8c92-c4d49af1499e" (UID: "68b2d21f-effb-4929-8c92-c4d49af1499e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:18.784837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.784659    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-xd8tc"]
Nov 03 10:51:18.857545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.857508    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.857545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.857544    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.857809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.857561    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b2d21f-effb-4929-8c92-c4d49af1499e-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.857809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.857594    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vn5\" (UniqueName: \"kubernetes.io/projected/68b2d21f-effb-4929-8c92-c4d49af1499e-kube-api-access-r2vn5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:18.946947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:18.946908    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:19.009862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.009703    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"]
Nov 03 10:51:19.012433 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:51:19.012405    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf10ca75_8590_4713_9b5e_f2642b7a7154.slice/crio-6df7c97a1570c14bccce859c3f7cf9213baaf7145826e0da05a936a36aadd15c WatchSource:0}: Error finding container 6df7c97a1570c14bccce859c3f7cf9213baaf7145826e0da05a936a36aadd15c: Status 404 returned error can't find the container with id 6df7c97a1570c14bccce859c3f7cf9213baaf7145826e0da05a936a36aadd15c
Nov 03 10:51:19.430335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.430305    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"
Nov 03 10:51:19.430335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.430323    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt" event={"ID":"68b2d21f-effb-4929-8c92-c4d49af1499e","Type":"ContainerDied","Data":"8415c2fc7f1e0c729e54f29647cd60dfbdbc8b5024cc7bbe96decaeb861d5364"}
Nov 03 10:51:19.430880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.430393    5817 scope.go:122] "RemoveContainer" containerID="fe98a4cd90dcfdcb7f742271acbb44ce3e67139f2de810783a1f660a51fb2e06"
Nov 03 10:51:19.432364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.432338    5817 generic.go:356] "Generic (PLEG): container finished" podID="a7f154d8-4d61-4d9b-bc60-92a3ef70fc03" containerID="083bb9b07b4ebf000df0979a3ff9772b143a77218ba8a874be4b5e723c160cb0" exitCode=0
Nov 03 10:51:19.432498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.432375    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5" event={"ID":"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03","Type":"ContainerDied","Data":"083bb9b07b4ebf000df0979a3ff9772b143a77218ba8a874be4b5e723c160cb0"}
Nov 03 10:51:19.435484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.435453    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl" event={"ID":"df10ca75-8590-4713-9b5e-f2642b7a7154","Type":"ContainerStarted","Data":"62a11d5f393d8f7cf9072138ad80d1bd00c6745ee26c46656d02ef70a4bf44a9"}
Nov 03 10:51:19.435616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.435492    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl" event={"ID":"df10ca75-8590-4713-9b5e-f2642b7a7154","Type":"ContainerStarted","Data":"6df7c97a1570c14bccce859c3f7cf9213baaf7145826e0da05a936a36aadd15c"}
Nov 03 10:51:19.462601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.462530    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"]
Nov 03 10:51:19.468612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.468592    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-wccjt"]
Nov 03 10:51:19.479871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.479834    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl" podStartSLOduration=1.479824803 podStartE2EDuration="1.479824803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:19.477873512 +0000 UTC m=+1309.683455574" watchObservedRunningTime="2026-11-03 10:51:19.479824803 +0000 UTC m=+1309.685406845"
Nov 03 10:51:19.869323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.869291    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:19.913474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.913443    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1541c102-d0ff-41af-805a-a15bc6951759" path="/var/lib/kubelet/pods/1541c102-d0ff-41af-805a-a15bc6951759/volumes"
Nov 03 10:51:19.914361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:19.914341    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" path="/var/lib/kubelet/pods/68b2d21f-effb-4929-8c92-c4d49af1499e/volumes"
Nov 03 10:51:20.445940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.445898    5817 generic.go:356] "Generic (PLEG): container finished" podID="df10ca75-8590-4713-9b5e-f2642b7a7154" containerID="62a11d5f393d8f7cf9072138ad80d1bd00c6745ee26c46656d02ef70a4bf44a9" exitCode=0
Nov 03 10:51:20.446346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.445984    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl" event={"ID":"df10ca75-8590-4713-9b5e-f2642b7a7154","Type":"ContainerDied","Data":"62a11d5f393d8f7cf9072138ad80d1bd00c6745ee26c46656d02ef70a4bf44a9"}
Nov 03 10:51:20.653222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.653193    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:20.658021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.657989    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:20.811338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.811255    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-openshift-ca\") pod \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") "
Nov 03 10:51:20.811512 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.811398    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-xks-ca\") pod \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") "
Nov 03 10:51:20.811667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.811649    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-kube-api-access-ggdm9\" (UniqueName: \"kubernetes.io/projected/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-kube-api-access-ggdm9\") pod \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\" (UID: \"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03\") "
Nov 03 10:51:20.811785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.811647    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-openshift-ca" pod "a7f154d8-4d61-4d9b-bc60-92a3ef70fc03" (UID: "a7f154d8-4d61-4d9b-bc60-92a3ef70fc03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:20.812217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.812198    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:20.813832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.813805    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-xks-ca" pod "a7f154d8-4d61-4d9b-bc60-92a3ef70fc03" (UID: "a7f154d8-4d61-4d9b-bc60-92a3ef70fc03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:51:20.813978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.813959    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-kube-api-access-ggdm9" pod "a7f154d8-4d61-4d9b-bc60-92a3ef70fc03" (UID: "a7f154d8-4d61-4d9b-bc60-92a3ef70fc03"). InnerVolumeSpecName "kube-api-access-ggdm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:20.913268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.913239    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:20.913268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:20.913265    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggdm9\" (UniqueName: \"kubernetes.io/projected/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03-kube-api-access-ggdm9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:21.442857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.442711    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:21.459424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.459395    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"
Nov 03 10:51:21.459424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.459398    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5" event={"ID":"a7f154d8-4d61-4d9b-bc60-92a3ef70fc03","Type":"ContainerDied","Data":"ad8783911a441d790c3723a1ca53c9ffbc8c7026fdabe3b5cb487ab61df7bc3e"}
Nov 03 10:51:21.459896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.459456    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad8783911a441d790c3723a1ca53c9ffbc8c7026fdabe3b5cb487ab61df7bc3e"
Nov 03 10:51:21.667733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.667706    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:21.828285 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.828212    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-openshift-ca\") pod \"df10ca75-8590-4713-9b5e-f2642b7a7154\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") "
Nov 03 10:51:21.828438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.828313    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-xks-ca\") pod \"df10ca75-8590-4713-9b5e-f2642b7a7154\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") "
Nov 03 10:51:21.828438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.828400    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df10ca75-8590-4713-9b5e-f2642b7a7154-kube-api-access-z9rjd\" (UniqueName: \"kubernetes.io/projected/df10ca75-8590-4713-9b5e-f2642b7a7154-kube-api-access-z9rjd\") pod \"df10ca75-8590-4713-9b5e-f2642b7a7154\" (UID: \"df10ca75-8590-4713-9b5e-f2642b7a7154\") "
Nov 03 10:51:21.828687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.828642    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-openshift-ca" pod "df10ca75-8590-4713-9b5e-f2642b7a7154" (UID: "df10ca75-8590-4713-9b5e-f2642b7a7154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:21.829329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.829307    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:21.830894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.830864    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-xks-ca" pod "df10ca75-8590-4713-9b5e-f2642b7a7154" (UID: "df10ca75-8590-4713-9b5e-f2642b7a7154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:51:21.831028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.831011    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df10ca75-8590-4713-9b5e-f2642b7a7154-kube-api-access-z9rjd" pod "df10ca75-8590-4713-9b5e-f2642b7a7154" (UID: "df10ca75-8590-4713-9b5e-f2642b7a7154"). InnerVolumeSpecName "kube-api-access-z9rjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:21.931023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.930993    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9rjd\" (UniqueName: \"kubernetes.io/projected/df10ca75-8590-4713-9b5e-f2642b7a7154-kube-api-access-z9rjd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:21.931140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:21.931030    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df10ca75-8590-4713-9b5e-f2642b7a7154-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:22.407933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:22.407882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:22.432735 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:22.432554    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:51:22.470852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:22.470819    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"
Nov 03 10:51:22.471343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:22.470822    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl" event={"ID":"df10ca75-8590-4713-9b5e-f2642b7a7154","Type":"ContainerDied","Data":"6df7c97a1570c14bccce859c3f7cf9213baaf7145826e0da05a936a36aadd15c"}
Nov 03 10:51:22.471343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:22.470945    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df7c97a1570c14bccce859c3f7cf9213baaf7145826e0da05a936a36aadd15c"
Nov 03 10:51:23.127506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.127438    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:51:23.155504 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.155469    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:23.255928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.255896    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-certs\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.255974    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-istio-token\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-istio-token\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256034    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-data\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256101    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-socket\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256393 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256138    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a28440-6ec6-432e-9154-a0ad0692c771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a28440-6ec6-432e-9154-a0ad0692c771-istiod-ca-cert\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256393 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256164    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-certs" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:23.256393 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256172    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00a28440-6ec6-432e-9154-a0ad0692c771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a28440-6ec6-432e-9154-a0ad0692c771-istio-podinfo\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256393 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256381    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-credential-socket\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256359    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-data" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:23.256692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256439    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-kube-api-access-5dt5f\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-kube-api-access-5dt5f\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256420    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-socket" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:23.256692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256506    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-envoy\") pod \"00a28440-6ec6-432e-9154-a0ad0692c771\" (UID: \"00a28440-6ec6-432e-9154-a0ad0692c771\") "
Nov 03 10:51:23.256692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256626    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-credential-socket" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:23.256692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.256549    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a28440-6ec6-432e-9154-a0ad0692c771-istiod-ca-cert" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:23.257131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.257100    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.257131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.257123    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.257302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.257141    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.257302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.257154    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.257302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.257168    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a28440-6ec6-432e-9154-a0ad0692c771-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.258422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.258402    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-istio-token" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:23.258906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.258884    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-envoy" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:23.259101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.259079    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00a28440-6ec6-432e-9154-a0ad0692c771-istio-podinfo" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:23.259152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.259098    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-kube-api-access-5dt5f" pod "00a28440-6ec6-432e-9154-a0ad0692c771" (UID: "00a28440-6ec6-432e-9154-a0ad0692c771"). InnerVolumeSpecName "kube-api-access-5dt5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:23.358676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.358652    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a28440-6ec6-432e-9154-a0ad0692c771-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.358676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.358676    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dt5f\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-kube-api-access-5dt5f\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.358835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.358692    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a28440-6ec6-432e-9154-a0ad0692c771-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.358835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.358703    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a28440-6ec6-432e-9154-a0ad0692c771-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:23.481885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.481851    5817 generic.go:356] "Generic (PLEG): container finished" podID="00a28440-6ec6-432e-9154-a0ad0692c771" containerID="a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5" exitCode=0
Nov 03 10:51:23.482333 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.481919    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"
Nov 03 10:51:23.482333 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.481925    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" event={"ID":"00a28440-6ec6-432e-9154-a0ad0692c771","Type":"ContainerDied","Data":"a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5"}
Nov 03 10:51:23.482333 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.481992    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl" event={"ID":"00a28440-6ec6-432e-9154-a0ad0692c771","Type":"ContainerDied","Data":"395a249f3f3cf67f6c15f0d5123ec77bba20c35a9b12dc232a8161db0922506d"}
Nov 03 10:51:23.482333 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.482019    5817 scope.go:122] "RemoveContainer" containerID="a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5"
Nov 03 10:51:23.496271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.496256    5817 scope.go:122] "RemoveContainer" containerID="7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc"
Nov 03 10:51:23.507681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.507658    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"]
Nov 03 10:51:23.509467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.509437    5817 scope.go:122] "RemoveContainer" containerID="a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5"
Nov 03 10:51:23.509826 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:23.509801    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5\": container with ID starting with a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5 not found: ID does not exist" containerID="a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5"
Nov 03 10:51:23.509910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.509836    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5"} err="failed to get container status \"a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5\": rpc error: code = NotFound desc = could not find container \"a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5\": container with ID starting with a79499549e8c0da6ba9b7e2c7d0a21f92a9f041eeef6d1c93104af9fa0b150d5 not found: ID does not exist"
Nov 03 10:51:23.509910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.509860    5817 scope.go:122] "RemoveContainer" containerID="7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc"
Nov 03 10:51:23.510146 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:23.510127    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc\": container with ID starting with 7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc not found: ID does not exist" containerID="7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc"
Nov 03 10:51:23.510205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.510158    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc"} err="failed to get container status \"7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc\": rpc error: code = NotFound desc = could not find container \"7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc\": container with ID starting with 7e37a00566c3791def17a9f93a98f8d659b3827bcebbcabc8d9a8eba60cd05cc not found: ID does not exist"
Nov 03 10:51:23.511765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.511745    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-de30a0-openshift-default-7845bcdbfc-7hvgl"]
Nov 03 10:51:23.846559 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.846462    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:23.907983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:23.907949    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" path="/var/lib/kubelet/pods/00a28440-6ec6-432e-9154-a0ad0692c771/volumes"
Nov 03 10:51:24.191756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:24.191728    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:51:24.739947 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:24.735828    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:25.747890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:25.747843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:26.386240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:26.386200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:27.564427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:27.564388    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:28.476887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:28.476563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:29.051854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:29.051810    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:29.181836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:29.181802    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:29.324517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:29.324428    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:29.473338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:29.473304    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:29.567730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:29.567698    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"]
Nov 03 10:51:29.572938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:29.572910    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2697c-wl6vl"]
Nov 03 10:51:29.905907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:29.905878    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df10ca75-8590-4713-9b5e-f2642b7a7154" path="/var/lib/kubelet/pods/df10ca75-8590-4713-9b5e-f2642b7a7154/volumes"
Nov 03 10:51:29.929323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:29.929295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:30.058691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:30.058634    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:30.836189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:30.836159    5817 scope.go:122] "RemoveContainer" containerID="a03b70032e6ce130125dbe503453e2acc6ac7178b5c1bb9db7d41549bd55568e"
Nov 03 10:51:30.846875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:30.846856    5817 scope.go:122] "RemoveContainer" containerID="fb8da1ab2fe5e7d6210820d78433278cf0baf905777ab20d8d08e22a0005737d"
Nov 03 10:51:30.856690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:30.856599    5817 scope.go:122] "RemoveContainer" containerID="a9ea7d21ee16ab5630cb353e6bc1be5ce87e86a8dc38bfa7800a61f094169f40"
Nov 03 10:51:30.866906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:30.866885    5817 scope.go:122] "RemoveContainer" containerID="31001e222ca2947932c0b048a18110f5a155a3a22eb56f63f44d840eebcfeea6"
Nov 03 10:51:30.979290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:30.977955    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:31.572281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:31.572097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:31.875255 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:31.873010    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:32.533154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:32.533120    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:32.668142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:32.668109    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:32.793790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:32.793521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:32.935041 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:32.935008    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:33.064768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.064682    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"]
Nov 03 10:51:33.065289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065273    5817 cpu_manager.go:414] "containerMap: removing container" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy"
Nov 03 10:51:33.065358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065289    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy"
Nov 03 10:51:33.065358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065301    5817 cpu_manager.go:414] "containerMap: removing container" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy"
Nov 03 10:51:33.065358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065305    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy"
Nov 03 10:51:33.065358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065332    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a7f154d8-4d61-4d9b-bc60-92a3ef70fc03" containerName="revoke-keys"
Nov 03 10:51:33.065358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065336    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f154d8-4d61-4d9b-bc60-92a3ef70fc03" containerName="revoke-keys"
Nov 03 10:51:33.065358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065355    5817 cpu_manager.go:414] "containerMap: removing container" podUID="df10ca75-8590-4713-9b5e-f2642b7a7154" containerName="revoke-keys"
Nov 03 10:51:33.065358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065360    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df10ca75-8590-4713-9b5e-f2642b7a7154" containerName="revoke-keys"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065376    5817 cpu_manager.go:414] "containerMap: removing container" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065381    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065388    5817 cpu_manager.go:414] "containerMap: removing container" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065392    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065473    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f154d8-4d61-4d9b-bc60-92a3ef70fc03" containerName="revoke-keys"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065483    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="df10ca75-8590-4713-9b5e-f2642b7a7154" containerName="revoke-keys"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065490    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065497    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a28440-6ec6-432e-9154-a0ad0692c771" containerName="istio-proxy"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065504    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b2d21f-effb-4929-8c92-c4d49af1499e" containerName="istio-proxy"
Nov 03 10:51:33.065744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.065510    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="1541c102-d0ff-41af-805a-a15bc6951759" containerName="istio-proxy"
Nov 03 10:51:33.071212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.071186    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.077521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.077496    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:51:33.077672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.077558    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:51:33.079454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.079432    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"]
Nov 03 10:51:33.146459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.146420    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.146690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.146588    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.146753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.146704    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t848\" (UniqueName: \"kubernetes.io/projected/2268cdc0-4a5a-467e-858e-9924436e84b0-kube-api-access-9t848\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.248316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.248271    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t848\" (UniqueName: \"kubernetes.io/projected/2268cdc0-4a5a-467e-858e-9924436e84b0-kube-api-access-9t848\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.248523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.248506    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.248650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.248633    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.249605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.249507    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.251329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.251306    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.272342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.272282    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t848\" (UniqueName: \"kubernetes.io/projected/2268cdc0-4a5a-467e-858e-9924436e84b0-kube-api-access-9t848\") pod \"maas-api-revoke-sub-81c8b50e-kmhpp\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.379402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.379308    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"]
Nov 03 10:51:33.382934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.382906    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4d9549-dsmm5"]
Nov 03 10:51:33.404840 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.404808    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:33.545188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.545155    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"]
Nov 03 10:51:33.554434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.554409    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.560046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.560019    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f81a79-openshift-default-dockercfg-gfsqg\""
Nov 03 10:51:33.560244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.560222    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"]
Nov 03 10:51:33.657913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.657830    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-token\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.657913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.657874    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-data\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.658082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.658004    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-certs\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.658082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.658038    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzjz\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-kube-api-access-6hzjz\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.658082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.658076    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-envoy\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.658198 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.658141    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-socket\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.658296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.658275    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-credential-socket\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.658338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.658314    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istiod-ca-cert\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.658383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.658368    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-podinfo\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.667340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.667307    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:33.691752 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:51:33.691726    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2268cdc0_4a5a_467e_858e_9924436e84b0.slice/crio-44896310bec3da0cae242fb88d77a2cd60d1281e3198cab81a95ee74fe11b372 WatchSource:0}: Error finding container 44896310bec3da0cae242fb88d77a2cd60d1281e3198cab81a95ee74fe11b372: Status 404 returned error can't find the container with id 44896310bec3da0cae242fb88d77a2cd60d1281e3198cab81a95ee74fe11b372
Nov 03 10:51:33.697168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.697147    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"]
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760295    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-socket\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760493    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-credential-socket\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760534    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istiod-ca-cert\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760593    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-podinfo\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760658    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-token\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760699    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-data\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760801    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-certs\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760824    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzjz\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-kube-api-access-6hzjz\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.761909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.760970    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-envoy\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.765320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.763361    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-socket\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.765320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.763645    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-credential-socket\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.765320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.764262    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istiod-ca-cert\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.765320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.764506    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-data\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.765320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.765036    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-certs\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.773968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.767680    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-envoy\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.785739 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.782019    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-token\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.785739 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.785122    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-podinfo\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.785739 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.785218    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzjz\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-kube-api-access-6hzjz\") pod \"e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") " pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.799614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.798320    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:33.886556 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.886525    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:33.907069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:33.907030    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f154d8-4d61-4d9b-bc60-92a3ef70fc03" path="/var/lib/kubelet/pods/a7f154d8-4d61-4d9b-bc60-92a3ef70fc03/volumes"
Nov 03 10:51:34.212845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.212816    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"]
Nov 03 10:51:34.213838 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:51:34.213815    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36b771b_d757_4cad_9ebf_12b74ed74bbe.slice/crio-80ace8d18dcabd3188dd314d25182de851ddd5b4e339664bd1ca80445bc24bba WatchSource:0}: Error finding container 80ace8d18dcabd3188dd314d25182de851ddd5b4e339664bd1ca80445bc24bba: Status 404 returned error can't find the container with id 80ace8d18dcabd3188dd314d25182de851ddd5b4e339664bd1ca80445bc24bba
Nov 03 10:51:34.217082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.217052    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:34.217166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.217116    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:34.217166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.217146    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:34.597826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.597281    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" event={"ID":"d36b771b-d757-4cad-9ebf-12b74ed74bbe","Type":"ContainerStarted","Data":"70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf"}
Nov 03 10:51:34.597826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.597342    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" event={"ID":"d36b771b-d757-4cad-9ebf-12b74ed74bbe","Type":"ContainerStarted","Data":"80ace8d18dcabd3188dd314d25182de851ddd5b4e339664bd1ca80445bc24bba"}
Nov 03 10:51:34.599261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.599227    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp" event={"ID":"2268cdc0-4a5a-467e-858e-9924436e84b0","Type":"ContainerStarted","Data":"f8ddd593ea37130828121ac14f73a990808398be308542d459705a7221d4970d"}
Nov 03 10:51:34.599387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.599282    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp" event={"ID":"2268cdc0-4a5a-467e-858e-9924436e84b0","Type":"ContainerStarted","Data":"44896310bec3da0cae242fb88d77a2cd60d1281e3198cab81a95ee74fe11b372"}
Nov 03 10:51:34.622428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.620349    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podStartSLOduration=1.6203335760000002 podStartE2EDuration="1.620333576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:34.616844908 +0000 UTC m=+1324.822426984" watchObservedRunningTime="2026-11-03 10:51:34.620333576 +0000 UTC m=+1324.825915627"
Nov 03 10:51:34.633355 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.633311    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp" podStartSLOduration=1.63329793 podStartE2EDuration="1.63329793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:34.630739059 +0000 UTC m=+1324.836321124" watchObservedRunningTime="2026-11-03 10:51:34.63329793 +0000 UTC m=+1324.838879981"
Nov 03 10:51:34.660756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.660726    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:34.887173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:34.887091    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:35.360745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:35.360702    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:35.609740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:35.609711    5817 generic.go:356] "Generic (PLEG): container finished" podID="2268cdc0-4a5a-467e-858e-9924436e84b0" containerID="f8ddd593ea37130828121ac14f73a990808398be308542d459705a7221d4970d" exitCode=0
Nov 03 10:51:35.609901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:35.609796    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp" event={"ID":"2268cdc0-4a5a-467e-858e-9924436e84b0","Type":"ContainerDied","Data":"f8ddd593ea37130828121ac14f73a990808398be308542d459705a7221d4970d"}
Nov 03 10:51:35.893027 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:35.892994    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:36.036899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:36.036872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:36.170399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:36.170281    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:36.397032 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:36.397000    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:36.621606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:36.620881    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:36.623088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:36.623064    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:51:36.879645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:36.879551    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:36.955090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:36.955014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:37.021756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.021720    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-xks-ca\") pod \"2268cdc0-4a5a-467e-858e-9924436e84b0\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") "
Nov 03 10:51:37.021933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.021891    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2268cdc0-4a5a-467e-858e-9924436e84b0-kube-api-access-9t848\" (UniqueName: \"kubernetes.io/projected/2268cdc0-4a5a-467e-858e-9924436e84b0-kube-api-access-9t848\") pod \"2268cdc0-4a5a-467e-858e-9924436e84b0\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") "
Nov 03 10:51:37.022022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.021972    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-openshift-ca\") pod \"2268cdc0-4a5a-467e-858e-9924436e84b0\" (UID: \"2268cdc0-4a5a-467e-858e-9924436e84b0\") "
Nov 03 10:51:37.025549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.025517    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-openshift-ca" pod "2268cdc0-4a5a-467e-858e-9924436e84b0" (UID: "2268cdc0-4a5a-467e-858e-9924436e84b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:37.026113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.026090    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-xks-ca" pod "2268cdc0-4a5a-467e-858e-9924436e84b0" (UID: "2268cdc0-4a5a-467e-858e-9924436e84b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:51:37.026242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.026163    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2268cdc0-4a5a-467e-858e-9924436e84b0-kube-api-access-9t848" pod "2268cdc0-4a5a-467e-858e-9924436e84b0" (UID: "2268cdc0-4a5a-467e-858e-9924436e84b0"). InnerVolumeSpecName "kube-api-access-9t848". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:37.124445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.124413    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t848\" (UniqueName: \"kubernetes.io/projected/2268cdc0-4a5a-467e-858e-9924436e84b0-kube-api-access-9t848\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:37.124445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.124442    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:37.124445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.124451    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2268cdc0-4a5a-467e-858e-9924436e84b0-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:37.644071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.644041    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"
Nov 03 10:51:37.645683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.645656    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp" event={"ID":"2268cdc0-4a5a-467e-858e-9924436e84b0","Type":"ContainerDied","Data":"44896310bec3da0cae242fb88d77a2cd60d1281e3198cab81a95ee74fe11b372"}
Nov 03 10:51:37.645865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.645848    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44896310bec3da0cae242fb88d77a2cd60d1281e3198cab81a95ee74fe11b372"
Nov 03 10:51:37.725596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.724836    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"]
Nov 03 10:51:37.729596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.725846    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2268cdc0-4a5a-467e-858e-9924436e84b0" containerName="revoke-keys"
Nov 03 10:51:37.729596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.725876    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2268cdc0-4a5a-467e-858e-9924436e84b0" containerName="revoke-keys"
Nov 03 10:51:37.729596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.726120    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2268cdc0-4a5a-467e-858e-9924436e84b0" containerName="revoke-keys"
Nov 03 10:51:37.732630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.730514    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.739396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.735520    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b3dc04-openshift-default-dockercfg-4cq8t\""
Nov 03 10:51:37.755004 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.752844    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"]
Nov 03 10:51:37.842128 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842088    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.842326 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842137    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.842326 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842301    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.842466 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842333    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.842535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842484    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.842668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842647    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd993708-fadc-48af-99eb-ca7bde93dfe9-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.842735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842688    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.842735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842729    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.842926 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.842753    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjm2k\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-kube-api-access-kjm2k\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.955345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.955302    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.955941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.955909    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.956071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.956032    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.956165 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.956074    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.956264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.956245    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.956428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.956406    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd993708-fadc-48af-99eb-ca7bde93dfe9-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.956619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.956560    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.956719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.956650    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.956719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.956693    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjm2k\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-kube-api-access-kjm2k\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.957898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.957871    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.959634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.958243    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd993708-fadc-48af-99eb-ca7bde93dfe9-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.959787 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.958322    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.959907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.958507    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.959907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.958969    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.961853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.961826    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.962531 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.962458    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.965458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.965431    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:37.966520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:37.966494    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjm2k\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-kube-api-access-kjm2k\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:38.092677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.092628    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:38.126727 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.126700    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:38.261839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.261712    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:38.399637 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:51:38.399603    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd993708_fadc_48af_99eb_ca7bde93dfe9.slice/crio-f72ca91f5a3939d0247e018d1122c769a817b198d348b9e1d02d89941ed67cd6 WatchSource:0}: Error finding container f72ca91f5a3939d0247e018d1122c769a817b198d348b9e1d02d89941ed67cd6: Status 404 returned error can't find the container with id f72ca91f5a3939d0247e018d1122c769a817b198d348b9e1d02d89941ed67cd6
Nov 03 10:51:38.400343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.400316    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"]
Nov 03 10:51:38.402648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.402613    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:38.402746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.402678    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:38.402746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.402717    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:38.490622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.490590    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"]
Nov 03 10:51:38.497990 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.497951    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.525682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.519427    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"]
Nov 03 10:51:38.525682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.525348    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-835861-openshift-default-dockercfg-vl48x\""
Nov 03 10:51:38.565641 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.565548    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-podinfo\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.565892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.565870    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istiod-ca-cert\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.566126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.566111    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-data\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.566316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.566302    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-socket\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.566526 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.566507    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-credential-socket\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.566729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.566713    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-token\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.566937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.566922    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-envoy\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.567361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.567324    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwt7\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-kube-api-access-gmwt7\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.567552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.567537    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-certs\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.583918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.583833    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:38.656303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.656269    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" event={"ID":"fd993708-fadc-48af-99eb-ca7bde93dfe9","Type":"ContainerStarted","Data":"65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714"}
Nov 03 10:51:38.656502 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.656329    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" event={"ID":"fd993708-fadc-48af-99eb-ca7bde93dfe9","Type":"ContainerStarted","Data":"f72ca91f5a3939d0247e018d1122c769a817b198d348b9e1d02d89941ed67cd6"}
Nov 03 10:51:38.672539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.672507    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-credential-socket\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.672692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.672560    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-token\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.672692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.672679    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-envoy\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.672790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.672731    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwt7\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-kube-api-access-gmwt7\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.672831 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.672809    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-certs\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.672923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.672906    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-podinfo\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.672981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.672941    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istiod-ca-cert\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.673042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.672944    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-credential-socket\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.673042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.673033    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-data\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.673155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.673107    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-socket\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.673240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.673207    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-certs\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.673383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.673364    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-data\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.673463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.673446    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-socket\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.673617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.673594    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istiod-ca-cert\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.676450 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.676048    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-podinfo\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.676450 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.676090    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-envoy\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.678882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.678685    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podStartSLOduration=1.67867264 podStartE2EDuration="1.67867264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:38.675843248 +0000 UTC m=+1328.881425300" watchObservedRunningTime="2026-11-03 10:51:38.67867264 +0000 UTC m=+1328.884254692"
Nov 03 10:51:38.681913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.681889    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-token\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.682358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.682335    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwt7\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-kube-api-access-gmwt7\") pod \"e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") " pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:38.742656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.742249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:38.844944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:38.844848    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:39.093444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.093409    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:39.095197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.095133    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 10:51:39.095314 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.095187    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 10:51:39.250662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.249436    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"]
Nov 03 10:51:39.251002 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:51:39.250974    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c59bd2_36ad_4d6b_a4b3_0bcbe488095f.slice/crio-4b511259c16d718dd2c67a7c1e37c31ee125c5b58a37bc169e4e9d67c472a491 WatchSource:0}: Error finding container 4b511259c16d718dd2c67a7c1e37c31ee125c5b58a37bc169e4e9d67c472a491: Status 404 returned error can't find the container with id 4b511259c16d718dd2c67a7c1e37c31ee125c5b58a37bc169e4e9d67c472a491
Nov 03 10:51:39.253546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.253516    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:39.253685 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.253598    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:39.253685 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.253643    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:39.412786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.412688    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:39.670508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.670408    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" event={"ID":"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f","Type":"ContainerStarted","Data":"8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376"}
Nov 03 10:51:39.670508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.670486    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" event={"ID":"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f","Type":"ContainerStarted","Data":"4b511259c16d718dd2c67a7c1e37c31ee125c5b58a37bc169e4e9d67c472a491"}
Nov 03 10:51:39.817252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.817208    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:39.846706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.846675    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:39.965683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:39.965645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:40.112596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.112022    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 10:51:40.112596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.112086    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 10:51:40.322781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.322684    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:40.808917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.806251    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podStartSLOduration=2.806230229 podStartE2EDuration="2.806230229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:39.740283415 +0000 UTC m=+1329.945865467" watchObservedRunningTime="2026-11-03 10:51:40.806230229 +0000 UTC m=+1331.011812281"
Nov 03 10:51:40.834716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.830513    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"]
Nov 03 10:51:40.837320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.835942    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:40.837320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.837120    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"]
Nov 03 10:51:40.841434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.841414    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:51:40.842816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.841720    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:51:40.855847 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.855826    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:40.978400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:40.978356    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:41.023801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.022740    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.023801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.022820    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.023801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.022918    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrg7\" (UniqueName: \"kubernetes.io/projected/6dc27193-08b1-4c7e-bc3e-291ac70be050-kube-api-access-lcrg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.119659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.118763    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 10:51:41.119659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.118819    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 10:51:41.124342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.124319    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.124473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.124368    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.124473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.124426    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrg7\" (UniqueName: \"kubernetes.io/projected/6dc27193-08b1-4c7e-bc3e-291ac70be050-kube-api-access-lcrg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.126076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.126051    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.127345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.127323    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.132229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.132207    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrg7\" (UniqueName: \"kubernetes.io/projected/6dc27193-08b1-4c7e-bc3e-291ac70be050-kube-api-access-lcrg7\") pod \"maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.158296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.158276    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:41.465214 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:51:41.465183    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc27193_08b1_4c7e_bc3e_291ac70be050.slice/crio-5d8db101fcc525b9d0a82129c4b07ae24f40f5593af12ac2ee9e984e230413ae WatchSource:0}: Error finding container 5d8db101fcc525b9d0a82129c4b07ae24f40f5593af12ac2ee9e984e230413ae: Status 404 returned error can't find the container with id 5d8db101fcc525b9d0a82129c4b07ae24f40f5593af12ac2ee9e984e230413ae
Nov 03 10:51:41.465909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.465886    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"]
Nov 03 10:51:41.696102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.696066    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f" event={"ID":"6dc27193-08b1-4c7e-bc3e-291ac70be050","Type":"ContainerStarted","Data":"5d8db101fcc525b9d0a82129c4b07ae24f40f5593af12ac2ee9e984e230413ae"}
Nov 03 10:51:41.697448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.697427    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:41.701855 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.701832    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:51:41.933140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:41.933110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:42.110650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.110298    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 10:51:42.110650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.110354    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 10:51:42.134655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.133254    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:42.268416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.268382    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:42.430654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.425108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:42.739352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.737287    5817 generic.go:356] "Generic (PLEG): container finished" podID="6dc27193-08b1-4c7e-bc3e-291ac70be050" containerID="4ac2651f268b43ad272a88b6c75660d3e3f3fffa1e9478b07ae2bc8a03df4dfb" exitCode=0
Nov 03 10:51:42.739352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.738618    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f" event={"ID":"6dc27193-08b1-4c7e-bc3e-291ac70be050","Type":"ContainerDied","Data":"4ac2651f268b43ad272a88b6c75660d3e3f3fffa1e9478b07ae2bc8a03df4dfb"}
Nov 03 10:51:42.976047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.976015    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"]
Nov 03 10:51:42.976252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.976231    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy" containerID="cri-o://22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d" gracePeriod=30
Nov 03 10:51:42.979971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.979950    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"]
Nov 03 10:51:42.980146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:42.980126    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy" containerID="cri-o://d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a" gracePeriod=30
Nov 03 10:51:43.107125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:43.106623    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 10:51:43.107125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:43.106694    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 10:51:43.379104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:43.379024    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2f7kn_99ada003-4afe-4019-8dab-6c9dadf4b888/payload-processing/0.log"
Nov 03 10:51:43.438347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:43.438291    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:43.598682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:43.598365    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2f7kn_99ada003-4afe-4019-8dab-6c9dadf4b888/payload-processing/0.log"
Nov 03 10:51:43.598682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:43.598627    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:43.963631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:43.960752    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:44.136598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.133996    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 10:51:44.136598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.134050    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 10:51:44.142099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.142071    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:44.163710 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.163670    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-openshift-ca\") pod \"6dc27193-08b1-4c7e-bc3e-291ac70be050\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") "
Nov 03 10:51:44.163956 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.163922    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-openshift-ca" pod "6dc27193-08b1-4c7e-bc3e-291ac70be050" (UID: "6dc27193-08b1-4c7e-bc3e-291ac70be050"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:44.164025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.163942    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc27193-08b1-4c7e-bc3e-291ac70be050-kube-api-access-lcrg7\" (UniqueName: \"kubernetes.io/projected/6dc27193-08b1-4c7e-bc3e-291ac70be050-kube-api-access-lcrg7\") pod \"6dc27193-08b1-4c7e-bc3e-291ac70be050\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") "
Nov 03 10:51:44.164068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.164040    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-xks-ca\") pod \"6dc27193-08b1-4c7e-bc3e-291ac70be050\" (UID: \"6dc27193-08b1-4c7e-bc3e-291ac70be050\") "
Nov 03 10:51:44.164705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.164680    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:44.166242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.166214    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc27193-08b1-4c7e-bc3e-291ac70be050-kube-api-access-lcrg7" pod "6dc27193-08b1-4c7e-bc3e-291ac70be050" (UID: "6dc27193-08b1-4c7e-bc3e-291ac70be050"). InnerVolumeSpecName "kube-api-access-lcrg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:44.166631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.166612    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-xks-ca" pod "6dc27193-08b1-4c7e-bc3e-291ac70be050" (UID: "6dc27193-08b1-4c7e-bc3e-291ac70be050"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:51:44.266129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.266094    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrg7\" (UniqueName: \"kubernetes.io/projected/6dc27193-08b1-4c7e-bc3e-291ac70be050-kube-api-access-lcrg7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:44.266129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.266123    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc27193-08b1-4c7e-bc3e-291ac70be050-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:44.824677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.824368    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f" event={"ID":"6dc27193-08b1-4c7e-bc3e-291ac70be050","Type":"ContainerDied","Data":"5d8db101fcc525b9d0a82129c4b07ae24f40f5593af12ac2ee9e984e230413ae"}
Nov 03 10:51:44.824677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.824532    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d8db101fcc525b9d0a82129c4b07ae24f40f5593af12ac2ee9e984e230413ae"
Nov 03 10:51:44.824677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.824471    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"
Nov 03 10:51:44.878473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.878445    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"]
Nov 03 10:51:44.879056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.879042    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6dc27193-08b1-4c7e-bc3e-291ac70be050" containerName="revoke-keys"
Nov 03 10:51:44.879099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.879058    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc27193-08b1-4c7e-bc3e-291ac70be050" containerName="revoke-keys"
Nov 03 10:51:44.879192 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.879184    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc27193-08b1-4c7e-bc3e-291ac70be050" containerName="revoke-keys"
Nov 03 10:51:44.882941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.882921    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:44.910390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.909068    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"]
Nov 03 10:51:44.960315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.960288    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:51:44.960617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.960600    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:51:44.975834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.975805    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxrz\" (UniqueName: \"kubernetes.io/projected/bab47064-2fad-4a30-8b22-b6163ce52383-kube-api-access-mfxrz\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:44.975974 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.975838    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:44.976065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:44.976035    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:45.078982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.077520    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxrz\" (UniqueName: \"kubernetes.io/projected/bab47064-2fad-4a30-8b22-b6163ce52383-kube-api-access-mfxrz\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:45.078982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.077610    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:45.078982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.077693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:45.078982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.077779    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:45.078982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.078808    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:45.086488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.081711    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:45.094503 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.094473    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxrz\" (UniqueName: \"kubernetes.io/projected/bab47064-2fad-4a30-8b22-b6163ce52383-kube-api-access-mfxrz\") pod \"maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:45.208928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.208890    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:45.212087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.212012    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:45.212833 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.212801    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:45.213366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.213343    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:51:45.529767 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:51:45.529736    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab47064_2fad_4a30_8b22_b6163ce52383.slice/crio-9106ab23baf8081245d7d64785e91712f390168ef083ae906e29d01c96fcd5aa WatchSource:0}: Error finding container 9106ab23baf8081245d7d64785e91712f390168ef083ae906e29d01c96fcd5aa: Status 404 returned error can't find the container with id 9106ab23baf8081245d7d64785e91712f390168ef083ae906e29d01c96fcd5aa
Nov 03 10:51:45.533481 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.533447    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"]
Nov 03 10:51:45.751693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.751661    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"]
Nov 03 10:51:45.752248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.751879    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy" containerID="cri-o://792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459" gracePeriod=30
Nov 03 10:51:45.769821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.769661    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"]
Nov 03 10:51:45.769951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.769896    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy" containerID="cri-o://6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb" gracePeriod=30
Nov 03 10:51:45.836513 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:45.836477    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj" event={"ID":"bab47064-2fad-4a30-8b22-b6163ce52383","Type":"ContainerStarted","Data":"9106ab23baf8081245d7d64785e91712f390168ef083ae906e29d01c96fcd5aa"}
Nov 03 10:51:46.418086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:46.417811    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:46.686510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:46.686483    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:46.846950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:46.846921    5817 generic.go:356] "Generic (PLEG): container finished" podID="bab47064-2fad-4a30-8b22-b6163ce52383" containerID="685229cffa35ab3a881b6f8ee50d573bd1adbab6ad632732d55f3ed654b6221d" exitCode=0
Nov 03 10:51:46.847332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:46.847000    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj" event={"ID":"bab47064-2fad-4a30-8b22-b6163ce52383","Type":"ContainerDied","Data":"685229cffa35ab3a881b6f8ee50d573bd1adbab6ad632732d55f3ed654b6221d"}
Nov 03 10:51:46.994088 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:46.994005    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:47.154606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:47.154180    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:48.072967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.071358    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:48.146991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.146963    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:48.243624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.239446    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:48.243624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.242936    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-openshift-ca\") pod \"bab47064-2fad-4a30-8b22-b6163ce52383\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") "
Nov 03 10:51:48.243624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.243095    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-xks-ca\") pod \"bab47064-2fad-4a30-8b22-b6163ce52383\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") "
Nov 03 10:51:48.243624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.243177    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab47064-2fad-4a30-8b22-b6163ce52383-kube-api-access-mfxrz\" (UniqueName: \"kubernetes.io/projected/bab47064-2fad-4a30-8b22-b6163ce52383-kube-api-access-mfxrz\") pod \"bab47064-2fad-4a30-8b22-b6163ce52383\" (UID: \"bab47064-2fad-4a30-8b22-b6163ce52383\") "
Nov 03 10:51:48.244513 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.244472    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-openshift-ca" pod "bab47064-2fad-4a30-8b22-b6163ce52383" (UID: "bab47064-2fad-4a30-8b22-b6163ce52383"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:48.248168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.248141    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab47064-2fad-4a30-8b22-b6163ce52383-kube-api-access-mfxrz" pod "bab47064-2fad-4a30-8b22-b6163ce52383" (UID: "bab47064-2fad-4a30-8b22-b6163ce52383"). InnerVolumeSpecName "kube-api-access-mfxrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:48.251044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.251019    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-xks-ca" pod "bab47064-2fad-4a30-8b22-b6163ce52383" (UID: "bab47064-2fad-4a30-8b22-b6163ce52383"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:51:48.440699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.359181    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.440699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.359227    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab47064-2fad-4a30-8b22-b6163ce52383-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.440699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.359242    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfxrz\" (UniqueName: \"kubernetes.io/projected/bab47064-2fad-4a30-8b22-b6163ce52383-kube-api-access-mfxrz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.440699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.422078    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:48.594611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.594535    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:51:48.598781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.598761    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:51:48.665018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.664993    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-token\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-token\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.665205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665094    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-token\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-token\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.665205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665162    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-certs\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.665331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665217    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-podinfo\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.665331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665255    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-socket\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.665435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665333    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-envoy\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.665435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665386    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-socket\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.665541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665437    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-envoy\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.665541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665502    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-kube-api-access-pn6gb\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-kube-api-access-pn6gb\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.665541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665510    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-certs" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.665718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665566    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f49022-14e6-474a-850e-def0cbe2e5a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f49022-14e6-474a-850e-def0cbe2e5a2-istiod-ca-cert\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.665718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665640    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-certs\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.665718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665670    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80241a2b-be47-424f-a5e3-4c96ef8e3802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80241a2b-be47-424f-a5e3-4c96ef8e3802-istiod-ca-cert\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.665873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665731    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-podinfo\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.665873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665772    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-data\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.665873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665776    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-socket" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.665873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665805    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-kube-api-access-t657n\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-kube-api-access-t657n\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.665873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665861    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-data\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.666117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665890    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-credential-socket\") pod \"80241a2b-be47-424f-a5e3-4c96ef8e3802\" (UID: \"80241a2b-be47-424f-a5e3-4c96ef8e3802\") "
Nov 03 10:51:48.666117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.665946    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-credential-socket\") pod \"22f49022-14e6-474a-850e-def0cbe2e5a2\" (UID: \"22f49022-14e6-474a-850e-def0cbe2e5a2\") "
Nov 03 10:51:48.666744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.666565    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-socket" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.666744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.666711    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-data" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.666951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.666875    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.666951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.666897    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.666951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.666911    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.667158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.667118    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-credential-socket" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.667349 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.667297    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-credential-socket" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.667683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.667461    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-data" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.667683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.667651    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-certs" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.667939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.667718    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f49022-14e6-474a-850e-def0cbe2e5a2-istiod-ca-cert" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:48.668087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.668053    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80241a2b-be47-424f-a5e3-4c96ef8e3802-istiod-ca-cert" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:48.668937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.668911    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-token" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:48.669045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.668986    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-kube-api-access-t657n" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "kube-api-access-t657n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:48.670759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.670737    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-podinfo" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:48.670868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.670849    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-podinfo" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:48.671109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.671084    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-kube-api-access-pn6gb" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "kube-api-access-pn6gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:48.671236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.671218    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-token" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:48.671357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.671334    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-envoy" pod "80241a2b-be47-424f-a5e3-4c96ef8e3802" (UID: "80241a2b-be47-424f-a5e3-4c96ef8e3802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.671412 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.671396    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-envoy" pod "22f49022-14e6-474a-850e-def0cbe2e5a2" (UID: "22f49022-14e6-474a-850e-def0cbe2e5a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:48.768555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768484    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768507    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80241a2b-be47-424f-a5e3-4c96ef8e3802-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768517    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768526    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768536    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t657n\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-kube-api-access-t657n\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768545    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768553    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768562    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768586    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768595    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768603    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768612    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80241a2b-be47-424f-a5e3-4c96ef8e3802-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768620    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f49022-14e6-474a-850e-def0cbe2e5a2-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768629    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn6gb\" (UniqueName: \"kubernetes.io/projected/22f49022-14e6-474a-850e-def0cbe2e5a2-kube-api-access-pn6gb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.768921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.768637    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f49022-14e6-474a-850e-def0cbe2e5a2-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:48.865052 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.865022    5817 generic.go:356] "Generic (PLEG): container finished" podID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerID="22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d" exitCode=0
Nov 03 10:51:48.865210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.865072    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"
Nov 03 10:51:48.865210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.865103    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" event={"ID":"80241a2b-be47-424f-a5e3-4c96ef8e3802","Type":"ContainerDied","Data":"22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d"}
Nov 03 10:51:48.865210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.865156    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5" event={"ID":"80241a2b-be47-424f-a5e3-4c96ef8e3802","Type":"ContainerDied","Data":"4e6da093ba1fa861b94c6847204152c16f99129cf805a46764b8114e56610eb9"}
Nov 03 10:51:48.865210 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.865180    5817 scope.go:122] "RemoveContainer" containerID="22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d"
Nov 03 10:51:48.866812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.866786    5817 generic.go:356] "Generic (PLEG): container finished" podID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerID="d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a" exitCode=0
Nov 03 10:51:48.866812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.866816    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" event={"ID":"22f49022-14e6-474a-850e-def0cbe2e5a2","Type":"ContainerDied","Data":"d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a"}
Nov 03 10:51:48.866812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.866833    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"
Nov 03 10:51:48.866812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.866870    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw" event={"ID":"22f49022-14e6-474a-850e-def0cbe2e5a2","Type":"ContainerDied","Data":"c89260fe02be5051d8a422a4dc8b76f34d21e3dc305922486df5ee9b01d59018"}
Nov 03 10:51:48.868770 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.868725    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj" event={"ID":"bab47064-2fad-4a30-8b22-b6163ce52383","Type":"ContainerDied","Data":"9106ab23baf8081245d7d64785e91712f390168ef083ae906e29d01c96fcd5aa"}
Nov 03 10:51:48.868880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.868793    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9106ab23baf8081245d7d64785e91712f390168ef083ae906e29d01c96fcd5aa"
Nov 03 10:51:48.870107 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.870083    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"
Nov 03 10:51:48.880149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.880135    5817 scope.go:122] "RemoveContainer" containerID="22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d"
Nov 03 10:51:48.880449 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:48.880428    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d\": container with ID starting with 22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d not found: ID does not exist" containerID="22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d"
Nov 03 10:51:48.880549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.880454    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d"} err="failed to get container status \"22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d\": rpc error: code = NotFound desc = could not find container \"22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d\": container with ID starting with 22ec5b66b3aa32b26c4e9bc23177fc255a93eb8f0b4db22de3860a09b9d6369d not found: ID does not exist"
Nov 03 10:51:48.880549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.880472    5817 scope.go:122] "RemoveContainer" containerID="d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a"
Nov 03 10:51:48.909698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.909650    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"]
Nov 03 10:51:48.913854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.913830    5817 scope.go:122] "RemoveContainer" containerID="d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a"
Nov 03 10:51:48.914241 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:48.914214    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a\": container with ID starting with d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a not found: ID does not exist" containerID="d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a"
Nov 03 10:51:48.914337 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.914252    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a"} err="failed to get container status \"d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a\": rpc error: code = NotFound desc = could not find container \"d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a\": container with ID starting with d4f50bc774e1a017073231ae57ab08f28501164e0bb2a33d0a459f21bf87780a not found: ID does not exist"
Nov 03 10:51:48.916037 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.916014    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-zrdsw"]
Nov 03 10:51:48.928953 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.928926    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"]
Nov 03 10:51:48.934473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:48.934453    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6nfm5"]
Nov 03 10:51:49.057674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:49.045725    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:49.439739 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:49.439711    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:49.903433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:49.903362    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" path="/var/lib/kubelet/pods/22f49022-14e6-474a-850e-def0cbe2e5a2/volumes"
Nov 03 10:51:49.903889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:49.903873    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" path="/var/lib/kubelet/pods/80241a2b-be47-424f-a5e3-4c96ef8e3802/volumes"
Nov 03 10:51:51.155121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.155094    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:51:51.184746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.184724    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:51:51.216902 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.216857    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-data\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.217077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.216914    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-kube-api-access-vrbkn\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-kube-api-access-vrbkn\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.217077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.217000    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-istio-token\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-istio-token\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.217077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.217070    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81954e6-e83d-44b4-876f-b5023399dc69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81954e6-e83d-44b4-876f-b5023399dc69-istiod-ca-cert\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.217305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.217157    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d81954e6-e83d-44b4-876f-b5023399dc69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81954e6-e83d-44b4-876f-b5023399dc69-istio-podinfo\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.217305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.217234    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-socket\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.217428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.217366    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-credential-socket\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.217428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.217409    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-certs\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.217546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.217462    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-envoy\") pod \"d81954e6-e83d-44b4-876f-b5023399dc69\" (UID: \"d81954e6-e83d-44b4-876f-b5023399dc69\") "
Nov 03 10:51:51.218667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.218623    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81954e6-e83d-44b4-876f-b5023399dc69-istiod-ca-cert" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:51.219045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.219019    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-certs" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.219143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.219082    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-credential-socket" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.219143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.219126    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-socket" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.219350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.219331    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-data" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.221715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.221695    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-envoy" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.221952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.221935    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-istio-token" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:51.222723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.222696    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d81954e6-e83d-44b4-876f-b5023399dc69-istio-podinfo" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:51.223800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.223776    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-kube-api-access-vrbkn" pod "d81954e6-e83d-44b4-876f-b5023399dc69" (UID: "d81954e6-e83d-44b4-876f-b5023399dc69"). InnerVolumeSpecName "kube-api-access-vrbkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.320056    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-kube-api-access-zdzkn\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-kube-api-access-zdzkn\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.320118    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-data\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.320252    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-istio-token\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-istio-token\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.320304    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7818f918-9435-469c-a3b6-c16e509f15ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7818f918-9435-469c-a3b6-c16e509f15ea-istio-podinfo\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.320423    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-envoy\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.320585    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-credential-socket\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.320848    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-credential-socket" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.321272    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7818f918-9435-469c-a3b6-c16e509f15ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7818f918-9435-469c-a3b6-c16e509f15ea-istiod-ca-cert\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.321347    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-certs\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.321648    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-socket\") pod \"7818f918-9435-469c-a3b6-c16e509f15ea\" (UID: \"7818f918-9435-469c-a3b6-c16e509f15ea\") "
Nov 03 10:51:51.322659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.322587    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7818f918-9435-469c-a3b6-c16e509f15ea-istiod-ca-cert" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:51.323469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.322832    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-certs" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.323469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.323231    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-socket" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.324159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324076    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-data" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.324794 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324536    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.324924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324827    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrbkn\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-kube-api-access-vrbkn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.324924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324848    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81954e6-e83d-44b4-876f-b5023399dc69-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.324924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324861    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81954e6-e83d-44b4-876f-b5023399dc69-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.324924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324876    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81954e6-e83d-44b4-876f-b5023399dc69-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.324924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324905    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.324924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324919    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.325243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324937    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7818f918-9435-469c-a3b6-c16e509f15ea-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.325243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324952    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.325243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324966    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.325243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324994    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.325243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.325009    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.325243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.325024    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81954e6-e83d-44b4-876f-b5023399dc69-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.325243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.324968    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-envoy" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:51.325651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.325631    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-istio-token" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:51.326203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.326183    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7818f918-9435-469c-a3b6-c16e509f15ea-istio-podinfo" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:51.326654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.326635    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-kube-api-access-zdzkn" pod "7818f918-9435-469c-a3b6-c16e509f15ea" (UID: "7818f918-9435-469c-a3b6-c16e509f15ea"). InnerVolumeSpecName "kube-api-access-zdzkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:51.426519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.426453    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdzkn\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-kube-api-access-zdzkn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.426519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.426479    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.426519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.426489    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7818f918-9435-469c-a3b6-c16e509f15ea-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.426519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.426498    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7818f918-9435-469c-a3b6-c16e509f15ea-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.426519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.426507    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7818f918-9435-469c-a3b6-c16e509f15ea-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:51.501473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.501450    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:51.632923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.632888    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:51.772463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.772429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:51.895869 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.895819    5817 generic.go:356] "Generic (PLEG): container finished" podID="d81954e6-e83d-44b4-876f-b5023399dc69" containerID="792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459" exitCode=0
Nov 03 10:51:51.896029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.895990    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"
Nov 03 10:51:51.900506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.899734    5817 generic.go:356] "Generic (PLEG): container finished" podID="7818f918-9435-469c-a3b6-c16e509f15ea" containerID="6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb" exitCode=0
Nov 03 10:51:51.900506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.899870    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"
Nov 03 10:51:51.900506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.900377    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:51.905003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.904978    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" event={"ID":"d81954e6-e83d-44b4-876f-b5023399dc69","Type":"ContainerDied","Data":"792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459"}
Nov 03 10:51:51.905124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.905030    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp" event={"ID":"d81954e6-e83d-44b4-876f-b5023399dc69","Type":"ContainerDied","Data":"4845f858d5eb4ee9629db4d51264871461b62dbd3901f33c30b95c3b0c5201f5"}
Nov 03 10:51:51.905124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.905054    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" event={"ID":"7818f918-9435-469c-a3b6-c16e509f15ea","Type":"ContainerDied","Data":"6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb"}
Nov 03 10:51:51.905124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.905079    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr" event={"ID":"7818f918-9435-469c-a3b6-c16e509f15ea","Type":"ContainerDied","Data":"2564e5e98e78a078c1f797144219ff44b99901afe85e242d78a9e91169e93ffc"}
Nov 03 10:51:51.905989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.905107    5817 scope.go:122] "RemoveContainer" containerID="792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459"
Nov 03 10:51:51.932709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.932655    5817 scope.go:122] "RemoveContainer" containerID="792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459"
Nov 03 10:51:51.932999 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:51.932972    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459\": container with ID starting with 792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459 not found: ID does not exist" containerID="792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459"
Nov 03 10:51:51.933060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.933010    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459"} err="failed to get container status \"792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459\": rpc error: code = NotFound desc = could not find container \"792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459\": container with ID starting with 792dd1c6f34fc79c5c614e3e584aeb4d0fbd0f22697155135b6a6b1de9f0a459 not found: ID does not exist"
Nov 03 10:51:51.933060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.933034    5817 scope.go:122] "RemoveContainer" containerID="6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb"
Nov 03 10:51:51.944729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.943737    5817 scope.go:122] "RemoveContainer" containerID="6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb"
Nov 03 10:51:51.944729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.944267    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"]
Nov 03 10:51:51.945178 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:51.945159    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb\": container with ID starting with 6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb not found: ID does not exist" containerID="6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb"
Nov 03 10:51:51.945233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.945184    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb"} err="failed to get container status \"6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb\": rpc error: code = NotFound desc = could not find container \"6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb\": container with ID starting with 6bb5990887d78211e38dce7626ea8899cdaab91c664becbf7e1df73e9a09e6cb not found: ID does not exist"
Nov 03 10:51:51.949075 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.949054    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-wx5tp"]
Nov 03 10:51:51.959679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.959649    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"]
Nov 03 10:51:51.963034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:51.963016    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-dpmfr"]
Nov 03 10:51:52.415409 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:51:52.415371    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:51:52.667857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:52.667619    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:53.559143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:53.559105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:53.902289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:53.902212    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" path="/var/lib/kubelet/pods/7818f918-9435-469c-a3b6-c16e509f15ea/volumes"
Nov 03 10:51:53.902871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:53.902852    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" path="/var/lib/kubelet/pods/d81954e6-e83d-44b4-876f-b5023399dc69/volumes"
Nov 03 10:51:54.347262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:54.347227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:55.045065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:55.045027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:55.168230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:55.168185    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:55.300438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:55.300358    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:56.132077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:56.130362    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:56.842449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:56.842411    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:57.079186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:57.079150    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:57.780736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:57.779136    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:57.996634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:57.996593    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:51:58.530667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:58.530420    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:51:59.522596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:51:59.522228    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:00.363596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:00.361551    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:00.537597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:00.536651    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:52:00.681749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:00.681713    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:52:00.700601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:00.700489    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:52:00.769598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:00.767035    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:52:01.371593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.366850    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:01.503564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.503507    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"]
Nov 03 10:52:01.511930 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.511901    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-814dc0-sk42f"]
Nov 03 10:52:01.520820 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.520631    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:01.539589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.539526    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:52:01.546958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.546932    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:52:01.602272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.594971    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:52:01.672396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.672314    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:52:01.815465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.815387    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:01.913594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.909675    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc27193-08b1-4c7e-bc3e-291ac70be050" path="/var/lib/kubelet/pods/6dc27193-08b1-4c7e-bc3e-291ac70be050/volumes"
Nov 03 10:52:01.979603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:01.979556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:02.452230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452198    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"]
Nov 03 10:52:02.452845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452830    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy"
Nov 03 10:52:02.452908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452846    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy"
Nov 03 10:52:02.452908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452876    5817 cpu_manager.go:414] "containerMap: removing container" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy"
Nov 03 10:52:02.452908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452881    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy"
Nov 03 10:52:02.452908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452897    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bab47064-2fad-4a30-8b22-b6163ce52383" containerName="revoke-keys"
Nov 03 10:52:02.452908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452903    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab47064-2fad-4a30-8b22-b6163ce52383" containerName="revoke-keys"
Nov 03 10:52:02.453067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452916    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy"
Nov 03 10:52:02.453067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452921    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy"
Nov 03 10:52:02.453067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452930    5817 cpu_manager.go:414] "containerMap: removing container" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy"
Nov 03 10:52:02.453067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.452934    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy"
Nov 03 10:52:02.453067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.453050    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81954e6-e83d-44b4-876f-b5023399dc69" containerName="istio-proxy"
Nov 03 10:52:02.453067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.453064    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="80241a2b-be47-424f-a5e3-4c96ef8e3802" containerName="istio-proxy"
Nov 03 10:52:02.453236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.453071    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab47064-2fad-4a30-8b22-b6163ce52383" containerName="revoke-keys"
Nov 03 10:52:02.453236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.453077    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7818f918-9435-469c-a3b6-c16e509f15ea" containerName="istio-proxy"
Nov 03 10:52:02.453236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.453083    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f49022-14e6-474a-850e-def0cbe2e5a2" containerName="istio-proxy"
Nov 03 10:52:02.458613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.458566    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.462786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.462763    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:52:02.462909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.462839    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:52:02.463566 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.463540    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"]
Nov 03 10:52:02.556620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.556528    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.557131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.556708    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.557131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.556810    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rp2z\" (UniqueName: \"kubernetes.io/projected/647dd6bf-02ac-40e1-a22d-1528f1968467-kube-api-access-5rp2z\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.657813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.657787    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rp2z\" (UniqueName: \"kubernetes.io/projected/647dd6bf-02ac-40e1-a22d-1528f1968467-kube-api-access-5rp2z\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.658146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.658131    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.658277 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.658266    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.658939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.658915    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.660876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.660853    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.668611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.666356    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rp2z\" (UniqueName: \"kubernetes.io/projected/647dd6bf-02ac-40e1-a22d-1528f1968467-kube-api-access-5rp2z\") pod \"maas-api-revoke-sub-a2731157-k7jm9\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.776225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.776138    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:02.927804 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:02.927754    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:03.060179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:03.060151    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"]
Nov 03 10:52:03.060921 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:52:03.060892    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647dd6bf_02ac_40e1_a22d_1528f1968467.slice/crio-645fc476a9a1e286ad1d6c6a9d0886fbc934e478ac559b2925a929d06536faac WatchSource:0}: Error finding container 645fc476a9a1e286ad1d6c6a9d0886fbc934e478ac559b2925a929d06536faac: Status 404 returned error can't find the container with id 645fc476a9a1e286ad1d6c6a9d0886fbc934e478ac559b2925a929d06536faac
Nov 03 10:52:03.466641 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:03.466601    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9" event={"ID":"647dd6bf-02ac-40e1-a22d-1528f1968467","Type":"ContainerStarted","Data":"ae724f59885c04fee7338c2b96cb4a4315b4c8074279d068a9ff40730af979cc"}
Nov 03 10:52:03.466775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:03.466668    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9" event={"ID":"647dd6bf-02ac-40e1-a22d-1528f1968467","Type":"ContainerStarted","Data":"645fc476a9a1e286ad1d6c6a9d0886fbc934e478ac559b2925a929d06536faac"}
Nov 03 10:52:03.484134 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:03.484090    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9" podStartSLOduration=1.484077202 podStartE2EDuration="1.484077202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:03.481524919 +0000 UTC m=+1353.687106970" watchObservedRunningTime="2026-11-03 10:52:03.484077202 +0000 UTC m=+1353.689659508"
Nov 03 10:52:03.698253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:03.698198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:04.472886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:04.472855    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:04.485287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:04.485123    5817 generic.go:356] "Generic (PLEG): container finished" podID="647dd6bf-02ac-40e1-a22d-1528f1968467" containerID="ae724f59885c04fee7338c2b96cb4a4315b4c8074279d068a9ff40730af979cc" exitCode=0
Nov 03 10:52:04.485287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:04.485164    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9" event={"ID":"647dd6bf-02ac-40e1-a22d-1528f1968467","Type":"ContainerDied","Data":"ae724f59885c04fee7338c2b96cb4a4315b4c8074279d068a9ff40730af979cc"}
Nov 03 10:52:05.428538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.428471    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:05.675286 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.675261    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:05.807694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.807610    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-openshift-ca\") pod \"647dd6bf-02ac-40e1-a22d-1528f1968467\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") "
Nov 03 10:52:05.807827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.807710    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-xks-ca\") pod \"647dd6bf-02ac-40e1-a22d-1528f1968467\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") "
Nov 03 10:52:05.807886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.807843    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647dd6bf-02ac-40e1-a22d-1528f1968467-kube-api-access-5rp2z\" (UniqueName: \"kubernetes.io/projected/647dd6bf-02ac-40e1-a22d-1528f1968467-kube-api-access-5rp2z\") pod \"647dd6bf-02ac-40e1-a22d-1528f1968467\" (UID: \"647dd6bf-02ac-40e1-a22d-1528f1968467\") "
Nov 03 10:52:05.808042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.808018    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-openshift-ca" pod "647dd6bf-02ac-40e1-a22d-1528f1968467" (UID: "647dd6bf-02ac-40e1-a22d-1528f1968467"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:05.808395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.808373    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:05.809929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.809904    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-xks-ca" pod "647dd6bf-02ac-40e1-a22d-1528f1968467" (UID: "647dd6bf-02ac-40e1-a22d-1528f1968467"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:52:05.810058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.810038    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647dd6bf-02ac-40e1-a22d-1528f1968467-kube-api-access-5rp2z" pod "647dd6bf-02ac-40e1-a22d-1528f1968467" (UID: "647dd6bf-02ac-40e1-a22d-1528f1968467"). InnerVolumeSpecName "kube-api-access-5rp2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:05.910035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.910006    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647dd6bf-02ac-40e1-a22d-1528f1968467-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:05.910136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:05.910040    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rp2z\" (UniqueName: \"kubernetes.io/projected/647dd6bf-02ac-40e1-a22d-1528f1968467-kube-api-access-5rp2z\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:06.124872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:06.124768    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:06.510808 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:06.510778    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"
Nov 03 10:52:06.511272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:06.510779    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9" event={"ID":"647dd6bf-02ac-40e1-a22d-1528f1968467","Type":"ContainerDied","Data":"645fc476a9a1e286ad1d6c6a9d0886fbc934e478ac559b2925a929d06536faac"}
Nov 03 10:52:06.511272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:06.510898    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="645fc476a9a1e286ad1d6c6a9d0886fbc934e478ac559b2925a929d06536faac"
Nov 03 10:52:06.983072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:06.983034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:07.135213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:07.132795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:07.636901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:07.636865    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:07.806845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:07.806809    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:07.954271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:07.954244    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:08.143441 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:08.143408    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:08.660677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:08.651519    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:09.244466 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:09.241223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:09.844240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:09.844187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:10.564324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:10.564261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:10.826239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:10.826152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:11.238735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:11.238690    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:11.378764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:11.378730    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:12.096077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:12.096044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:12.754660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:12.754626    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:13.726846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:13.726779    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:15.022022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:15.021983    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:15.231094 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:15.231063    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:15.366298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:15.366192    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:15.869905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:15.869872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:16.467782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:16.467618    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:16.956640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:16.956608    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:17.322644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:17.318115    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:17.962360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:17.962327    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:18.353709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.353612    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:18.547633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.529477    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:18.684642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.679054    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"]
Nov 03 10:52:18.684642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.680104    5817 cpu_manager.go:414] "containerMap: removing container" podUID="647dd6bf-02ac-40e1-a22d-1528f1968467" containerName="revoke-keys"
Nov 03 10:52:18.684642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.680133    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647dd6bf-02ac-40e1-a22d-1528f1968467" containerName="revoke-keys"
Nov 03 10:52:18.684642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.680360    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="647dd6bf-02ac-40e1-a22d-1528f1968467" containerName="revoke-keys"
Nov 03 10:52:18.863124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.862889    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:18.885650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.885621    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"]
Nov 03 10:52:18.885880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.885782    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:18.947647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.936997    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"]
Nov 03 10:52:18.963150 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.960744    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-08c392-openshift-default-dockercfg-72pjm\""
Nov 03 10:52:18.990606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:18.985618    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a51d55-4pkfj"]
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132126    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istiod-ca-cert\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132189    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-credential-socket\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132228    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-data\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132263    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-certs\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132286    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-token\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132308    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-envoy\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132367    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-socket\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132432    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvf5l\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-kube-api-access-qvf5l\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.132610    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-podinfo\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.142269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.138961    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:19.252525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.252431    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istiod-ca-cert\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.253826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.253800    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-credential-socket\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.254039 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.254021    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-data\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.254178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.254164    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-certs\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.254291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.254278    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-token\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.254426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.254410    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-envoy\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.254605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.254588    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-socket\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.254805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.254786    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvf5l\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-kube-api-access-qvf5l\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.255027 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.255012    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-podinfo\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.265866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.255925    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-certs\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.265866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.253718    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istiod-ca-cert\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.265866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.256217    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-credential-socket\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.266985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.266931    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-data\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.267181 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.266944    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-socket\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.291003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.290920    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-envoy\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.328498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.327304    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvf5l\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-kube-api-access-qvf5l\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.328498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.327796    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-podinfo\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.336108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.336075    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-token\") pod \"e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") " pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.428598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.426607    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:19.467999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.467958    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:19.863905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.863873    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"]
Nov 03 10:52:19.867968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.867878    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:19.867968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.867942    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:19.868169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.867975    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:19.963222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:19.963197    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab47064-2fad-4a30-8b22-b6163ce52383" path="/var/lib/kubelet/pods/bab47064-2fad-4a30-8b22-b6163ce52383/volumes"
Nov 03 10:52:20.026469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:20.026437    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:20.277588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:20.276417    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" event={"ID":"c5173ad5-8182-4a4b-9bf0-a291aca5f363","Type":"ContainerStarted","Data":"aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18"}
Nov 03 10:52:20.277588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:20.276509    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" event={"ID":"c5173ad5-8182-4a4b-9bf0-a291aca5f363","Type":"ContainerStarted","Data":"bcf68a3fb1b9765057b67b0e67d5aca842baa94e4f531ef2962558490abfc82e"}
Nov 03 10:52:20.332949 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:20.332884    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" podStartSLOduration=2.33285834 podStartE2EDuration="2.33285834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:20.310913151 +0000 UTC m=+1370.516495202" watchObservedRunningTime="2026-11-03 10:52:20.33285834 +0000 UTC m=+1370.538440400"
Nov 03 10:52:20.487881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:20.487850    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:20.843606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:20.843450    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:21.469033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.468969    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:21.481770 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.481739    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:21.762323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.762286    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"]
Nov 03 10:52:21.768425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.768330    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"]
Nov 03 10:52:21.768649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.768464    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.780154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.776967    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:52:21.780154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.777164    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:52:21.826448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.826413    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.826675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.826529    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzc6r\" (UniqueName: \"kubernetes.io/projected/c49db8d6-9ce6-432c-a15d-abab988f49e0-kube-api-access-rzc6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.828762 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.827076    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.928605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.928548    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.928813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.928722    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.929763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.929730    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzc6r\" (UniqueName: \"kubernetes.io/projected/c49db8d6-9ce6-432c-a15d-abab988f49e0-kube-api-access-rzc6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.929901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.929858    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.932175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.932145    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:21.939250 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:21.938946    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzc6r\" (UniqueName: \"kubernetes.io/projected/c49db8d6-9ce6-432c-a15d-abab988f49e0-kube-api-access-rzc6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:22.123609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.123506    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:22.307593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.306730    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:22.308723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.308422    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:22.425800 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:52:22.425725    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:52:22.453292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.441263    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"]
Nov 03 10:52:22.457876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.457691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:22.665982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.665938    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:52:22.666461 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.666026    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:52:22.717184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.717079    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:22.785641 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.785533    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"]
Nov 03 10:52:22.792075 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.792050    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.801491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.801462    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"]
Nov 03 10:52:22.862629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.862549    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:22.869767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.869733    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.869919 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.869791    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zvb6\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-kube-api-access-2zvb6\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.869919 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.869832    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae1e29d-a609-43ca-b824-1851513008ac-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.870024 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.869922    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.870123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.870105    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.870202 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.870156    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.870268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.870238    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.870344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.870326    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.870401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.870359    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae1e29d-a609-43ca-b824-1851513008ac-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972079    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972138    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvb6\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-kube-api-access-2zvb6\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972163    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae1e29d-a609-43ca-b824-1851513008ac-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972189    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972283    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972331    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972373    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972448    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972477    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae1e29d-a609-43ca-b824-1851513008ac-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972817    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.972874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.972842    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.973062 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.973039    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae1e29d-a609-43ca-b824-1851513008ac-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.973117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.973074    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.973243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.973226    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.975016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.974991    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.975269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.975249    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae1e29d-a609-43ca-b824-1851513008ac-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.980731 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.980696    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zvb6\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-kube-api-access-2zvb6\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:22.980935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:22.980915    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:23.112779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.112745    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:23.253203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.253108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:23.365406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.335238    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl" event={"ID":"c49db8d6-9ce6-432c-a15d-abab988f49e0","Type":"ContainerStarted","Data":"fee993e3d2880683f45f5f61b53c380549f1a87fc77dc5204edb99b8f63c4104"}
Nov 03 10:52:23.372944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.369732    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl" event={"ID":"c49db8d6-9ce6-432c-a15d-abab988f49e0","Type":"ContainerStarted","Data":"33791af225baa29ff62cfbd0a09ef17f9b778cb07b25dd211bafa1752ca7d6a9"}
Nov 03 10:52:23.391391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.391087    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:23.405224 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.405165    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl" podStartSLOduration=2.405143954 podStartE2EDuration="2.405143954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:23.398641237 +0000 UTC m=+1373.604223284" watchObservedRunningTime="2026-11-03 10:52:23.405143954 +0000 UTC m=+1373.610726006"
Nov 03 10:52:23.438280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.438226    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"]
Nov 03 10:52:23.438555 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:52:23.438497    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae1e29d_a609_43ca_b824_1851513008ac.slice/crio-b93d9afeed4a9d071a0cfa7092b69031de6b6226ddaba144705c6d399f49d607 WatchSource:0}: Error finding container b93d9afeed4a9d071a0cfa7092b69031de6b6226ddaba144705c6d399f49d607: Status 404 returned error can't find the container with id b93d9afeed4a9d071a0cfa7092b69031de6b6226ddaba144705c6d399f49d607
Nov 03 10:52:23.447648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.441959    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:23.447648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.442050    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:23.447648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.442093    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:23.967912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:23.967824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:24.110722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.110671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:24.356237 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.356106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:24.395170 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.395135    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" event={"ID":"9ae1e29d-a609-43ca-b824-1851513008ac","Type":"ContainerStarted","Data":"2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55"}
Nov 03 10:52:24.407593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.407503    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" event={"ID":"9ae1e29d-a609-43ca-b824-1851513008ac","Type":"ContainerStarted","Data":"b93d9afeed4a9d071a0cfa7092b69031de6b6226ddaba144705c6d399f49d607"}
Nov 03 10:52:24.450316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.449120    5817 generic.go:356] "Generic (PLEG): container finished" podID="c49db8d6-9ce6-432c-a15d-abab988f49e0" containerID="fee993e3d2880683f45f5f61b53c380549f1a87fc77dc5204edb99b8f63c4104" exitCode=0
Nov 03 10:52:24.450316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.450153    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl" event={"ID":"c49db8d6-9ce6-432c-a15d-abab988f49e0","Type":"ContainerDied","Data":"fee993e3d2880683f45f5f61b53c380549f1a87fc77dc5204edb99b8f63c4104"}
Nov 03 10:52:24.470236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.469227    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podStartSLOduration=2.469205728 podStartE2EDuration="2.469205728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:24.454093625 +0000 UTC m=+1374.659675677" watchObservedRunningTime="2026-11-03 10:52:24.469205728 +0000 UTC m=+1374.674787781"
Nov 03 10:52:24.507948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.507913    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:24.954872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:24.950147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:25.116598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.113106    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:25.116598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.114762    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 10:52:25.116598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.114822    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 10:52:25.676864 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.676819    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:25.873007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.872979    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:25.881762 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.881741    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-xks-ca\") pod \"c49db8d6-9ce6-432c-a15d-abab988f49e0\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") "
Nov 03 10:52:25.881890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.881876    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-openshift-ca\") pod \"c49db8d6-9ce6-432c-a15d-abab988f49e0\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") "
Nov 03 10:52:25.882242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.881971    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49db8d6-9ce6-432c-a15d-abab988f49e0-kube-api-access-rzc6r\" (UniqueName: \"kubernetes.io/projected/c49db8d6-9ce6-432c-a15d-abab988f49e0-kube-api-access-rzc6r\") pod \"c49db8d6-9ce6-432c-a15d-abab988f49e0\" (UID: \"c49db8d6-9ce6-432c-a15d-abab988f49e0\") "
Nov 03 10:52:25.882242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.882195    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-openshift-ca" pod "c49db8d6-9ce6-432c-a15d-abab988f49e0" (UID: "c49db8d6-9ce6-432c-a15d-abab988f49e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:25.882834 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.882811    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:25.884247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.884220    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-xks-ca" pod "c49db8d6-9ce6-432c-a15d-abab988f49e0" (UID: "c49db8d6-9ce6-432c-a15d-abab988f49e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:52:25.884558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.884538    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49db8d6-9ce6-432c-a15d-abab988f49e0-kube-api-access-rzc6r" pod "c49db8d6-9ce6-432c-a15d-abab988f49e0" (UID: "c49db8d6-9ce6-432c-a15d-abab988f49e0"). InnerVolumeSpecName "kube-api-access-rzc6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:25.990736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.984891    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzc6r\" (UniqueName: \"kubernetes.io/projected/c49db8d6-9ce6-432c-a15d-abab988f49e0-kube-api-access-rzc6r\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:25.990736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:25.984941    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49db8d6-9ce6-432c-a15d-abab988f49e0-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:26.116110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.116011    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 10:52:26.116110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.116074    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 10:52:26.300601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.296558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:26.476599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.476146    5817 generic.go:356] "Generic (PLEG): container finished" podID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerID="70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf" exitCode=0
Nov 03 10:52:26.476599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.476357    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" event={"ID":"d36b771b-d757-4cad-9ebf-12b74ed74bbe","Type":"ContainerDied","Data":"70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf"}
Nov 03 10:52:26.477304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.476932    5817 scope.go:122] "RemoveContainer" containerID="70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf"
Nov 03 10:52:26.479027 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.478989    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:26.479333 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.479200    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:26.479535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.479512    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:26.482353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.482297    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl" event={"ID":"c49db8d6-9ce6-432c-a15d-abab988f49e0","Type":"ContainerDied","Data":"33791af225baa29ff62cfbd0a09ef17f9b778cb07b25dd211bafa1752ca7d6a9"}
Nov 03 10:52:26.482353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.482324    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"
Nov 03 10:52:26.482530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:26.482352    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33791af225baa29ff62cfbd0a09ef17f9b778cb07b25dd211bafa1752ca7d6a9"
Nov 03 10:52:27.114148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.114057    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 10:52:27.114148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.114132    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 10:52:27.161302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.160000    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:27.375459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.375233    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": read tcp 10.133.0.2:57216->10.133.0.144:15021: read: connection reset by peer" start-of-body=
Nov 03 10:52:27.375459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.375305    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": read tcp 10.133.0.2:57216->10.133.0.144:15021: read: connection reset by peer"
Nov 03 10:52:27.510599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.510228    5817 generic.go:356] "Generic (PLEG): container finished" podID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerID="8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376" exitCode=0
Nov 03 10:52:27.510599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.510361    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" event={"ID":"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f","Type":"ContainerDied","Data":"8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376"}
Nov 03 10:52:27.514598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.510904    5817 scope.go:122] "RemoveContainer" containerID="8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376"
Nov 03 10:52:27.552923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.540882    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:27.552923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.541390    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:27.552923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.545047    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:27.564308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.564278    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" event={"ID":"d36b771b-d757-4cad-9ebf-12b74ed74bbe","Type":"ContainerStarted","Data":"600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99"}
Nov 03 10:52:27.954595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.947763    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:52:27.954595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.948000    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 10:52:27.954595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.948050    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 10:52:27.968340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:27.962436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:28.116594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.115988    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 10:52:28.116594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.116067    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 10:52:28.578639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.578606    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" event={"ID":"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f","Type":"ContainerStarted","Data":"5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0"}
Nov 03 10:52:28.787246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.787210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:28.851490 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.851012    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:52:28.851490 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.851137    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 10:52:28.851490 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.851184    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 10:52:28.892600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.892547    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 10:52:28.892884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:28.892854    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 10:52:29.127655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:29.126894    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 10:52:29.127655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:29.126957    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 10:52:29.457915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:29.456228    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:29.851091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:29.850997    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 10:52:29.851712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:29.851674    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 10:52:29.936873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:29.935859    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 10:52:29.936873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:29.935916    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 10:52:30.060298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.060182    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:30.128646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.122144    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 10:52:30.128646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.122201    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 10:52:30.768609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.763938    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:30.861596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.856484    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 10:52:30.861596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.856540    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 10:52:30.901631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.892097    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 10:52:30.901631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.892158    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 10:52:30.923600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.915544    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:30.965598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.959921    5817 scope.go:122] "RemoveContainer" containerID="605e01e9136a5716a4a57e0a2ea6b6c48a2b1931e425ffb6a67753025239e92a"
Nov 03 10:52:30.984608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:30.984566    5817 scope.go:122] "RemoveContainer" containerID="4eab510d32c1262527dd6690989db7a2be1571efa04a0d4d66490eb9f85d21a4"
Nov 03 10:52:31.050493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.050429    5817 scope.go:122] "RemoveContainer" containerID="0370429461ae455fce349468e235e561bdf2b5251b4bc30b6070aeafd19e0494"
Nov 03 10:52:31.061564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.061544    5817 scope.go:122] "RemoveContainer" containerID="179bded08410c50380cce5e19b42f9f2367c249125984a7a158e9fcb0c1df110"
Nov 03 10:52:31.145018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.144977    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:31.145799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.145738    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:31.152160 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.149028    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:31.617562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.617523    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:31.852599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.850240    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 10:52:31.852599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.850298    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 10:52:31.924646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.910012    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 10:52:31.924646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:31.910070    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 10:52:32.058595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:32.058126    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:32.216593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:32.216288    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:32.359618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:32.356632    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:32.854598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:32.849393    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 10:52:32.854598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:32.849618    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 10:52:32.898624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:32.898261    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 10:52:32.898624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:32.898321    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 10:52:33.390618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:33.387745    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:33.848631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:33.847802    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 10:52:33.848631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:33.847875    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 10:52:33.919950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:33.919913    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:33.924828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:33.924802    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:52:33.925165 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:33.925145    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:52:33.926004 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:33.925989    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:52:34.040226 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:34.040153    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"]
Nov 03 10:52:34.043806 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:34.043779    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b3dc04-2hfzl"]
Nov 03 10:52:34.060717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:34.059992    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:34.213622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:34.213457    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:34.620359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:34.620274    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:34.911923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:34.911724    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:52:34.912911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:34.912829    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:52:34.913647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:34.913626    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:52:35.048238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:35.048202    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:35.364913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:35.364876    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:35.855824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:35.855779    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:35.919625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:35.918347    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49db8d6-9ce6-432c-a15d-abab988f49e0" path="/var/lib/kubelet/pods/c49db8d6-9ce6-432c-a15d-abab988f49e0/volumes"
Nov 03 10:52:36.515802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:36.515766    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:36.651183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:36.651147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:36.940687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:36.938311    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:37.105595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:37.102490    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:37.851471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:37.851186    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:38.572431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:38.572398    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:39.241385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:39.241354    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:40.150813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:40.150773    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:40.853539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:40.853505    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:41.016910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:41.016882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:41.146764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:41.146652    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"]
Nov 03 10:52:41.146957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:41.146929    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy" containerID="cri-o://2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55" gracePeriod=30
Nov 03 10:52:41.153675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:41.153650    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"]
Nov 03 10:52:41.153969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:41.153887    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy" containerID="cri-o://65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714" gracePeriod=30
Nov 03 10:52:41.170908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:41.170881    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:41.315493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:41.315448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:41.985621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:41.985591    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:42.155167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:42.153789    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:42.297306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:42.297213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:42.581793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:42.581655    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:42.633543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:42.633510    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:52:42.743746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:42.743705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:43.341090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:43.341057    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:44.253688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.253653    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:44.575482 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.575398    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"]
Nov 03 10:52:44.576314 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.576296    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c49db8d6-9ce6-432c-a15d-abab988f49e0" containerName="revoke-keys"
Nov 03 10:52:44.576373 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.576317    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49db8d6-9ce6-432c-a15d-abab988f49e0" containerName="revoke-keys"
Nov 03 10:52:44.576538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.576526    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49db8d6-9ce6-432c-a15d-abab988f49e0" containerName="revoke-keys"
Nov 03 10:52:44.585752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.585723    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"]
Nov 03 10:52:44.585932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.585852    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.604521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.604493    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:52:44.605822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.605801    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:52:44.747406 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.747375    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.747602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.747534    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.747766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.747742    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6d5n\" (UniqueName: \"kubernetes.io/projected/492cb988-75a8-4ca3-98a5-1d6584e88066-kube-api-access-q6d5n\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.848800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.848712    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.848957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.848838    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.848997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.848982    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6d5n\" (UniqueName: \"kubernetes.io/projected/492cb988-75a8-4ca3-98a5-1d6584e88066-kube-api-access-q6d5n\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.849424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.849402    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.851398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.851354    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.857690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.857668    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6d5n\" (UniqueName: \"kubernetes.io/projected/492cb988-75a8-4ca3-98a5-1d6584e88066-kube-api-access-q6d5n\") pod \"maas-api-revoke-keys-e2e-praxis-a-835861-9klhp\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:44.948729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:44.948690    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:45.048627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:45.042239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:45.224690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:45.224389    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:52:45.224690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:45.224438    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:52:45.284760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:45.284736    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"]
Nov 03 10:52:45.285799 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:52:45.285775    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492cb988_75a8_4ca3_98a5_1d6584e88066.slice/crio-aa7fff3370d3af2bf9f4bffee7f4ca151a97b3c983d73d4078d6adffb852d5b8 WatchSource:0}: Error finding container aa7fff3370d3af2bf9f4bffee7f4ca151a97b3c983d73d4078d6adffb852d5b8: Status 404 returned error can't find the container with id aa7fff3370d3af2bf9f4bffee7f4ca151a97b3c983d73d4078d6adffb852d5b8
Nov 03 10:52:45.769273 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:45.768971    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:45.980074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:45.979982    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp" event={"ID":"492cb988-75a8-4ca3-98a5-1d6584e88066","Type":"ContainerStarted","Data":"6c20a1487c323c797dabd921bbb85ce928e2e56a12e5839375938db7800a7e91"}
Nov 03 10:52:45.980074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:45.980045    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp" event={"ID":"492cb988-75a8-4ca3-98a5-1d6584e88066","Type":"ContainerStarted","Data":"aa7fff3370d3af2bf9f4bffee7f4ca151a97b3c983d73d4078d6adffb852d5b8"}
Nov 03 10:52:46.022597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.020017    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp" podStartSLOduration=2.019989189 podStartE2EDuration="2.019989189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:46.008993069 +0000 UTC m=+1396.214575121" watchObservedRunningTime="2026-11-03 10:52:46.019989189 +0000 UTC m=+1396.225571241"
Nov 03 10:52:46.147672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.147637    5817 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:52:46.147840 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.147700    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:52:46.542256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.542230    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:46.565148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.565125    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:52:46.583203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.583165    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-kube-api-access-2zvb6\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-kube-api-access-2zvb6\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.583353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.583262    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-kube-api-access-kjm2k\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-kube-api-access-kjm2k\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.583353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.583312    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-envoy\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.583472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.583356    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-podinfo\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.583472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.583425    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-credential-socket\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.583472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.583463    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-socket\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.583641 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.583495    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-data\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.583641 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.583528    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-istio-token\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.584433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.584109    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-credential-socket" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.584606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.584455    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-socket" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.584879    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae1e29d-a609-43ca-b824-1851513008ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae1e29d-a609-43ca-b824-1851513008ac-istiod-ca-cert\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.584920    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae1e29d-a609-43ca-b824-1851513008ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae1e29d-a609-43ca-b824-1851513008ac-istio-podinfo\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.584968    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-data" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585068    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd993708-fadc-48af-99eb-ca7bde93dfe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd993708-fadc-48af-99eb-ca7bde93dfe9-istiod-ca-cert\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585122    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-certs\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585163    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-envoy\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585216    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-credential-socket\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585269    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-token\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-token\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585303    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae1e29d-a609-43ca-b824-1851513008ac-istiod-ca-cert" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585334    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-certs\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585366    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-data\") pod \"9ae1e29d-a609-43ca-b824-1851513008ac\" (UID: \"9ae1e29d-a609-43ca-b824-1851513008ac\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585418    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-socket\") pod \"fd993708-fadc-48af-99eb-ca7bde93dfe9\" (UID: \"fd993708-fadc-48af-99eb-ca7bde93dfe9\") "
Nov 03 10:52:46.586138 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.585637    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-certs" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.632162    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd993708-fadc-48af-99eb-ca7bde93dfe9-istiod-ca-cert" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.632359    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-kube-api-access-kjm2k" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "kube-api-access-kjm2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.632461    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-kube-api-access-2zvb6" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "kube-api-access-2zvb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.632599    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-podinfo" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633207    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zvb6\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-kube-api-access-2zvb6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633231    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjm2k\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-kube-api-access-kjm2k\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633245    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633266    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633280    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633294    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633309    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae1e29d-a609-43ca-b824-1851513008ac-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633326    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd993708-fadc-48af-99eb-ca7bde93dfe9-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.633339    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.638383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.636275    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae1e29d-a609-43ca-b824-1851513008ac-istio-podinfo" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:52:46.638383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.637103    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-credential-socket" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.638383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.637348    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-certs" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.638383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.637528    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-data" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.639599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.639550    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-socket" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.684484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.672785    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-envoy" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.684484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.673402    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:46.684484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.673554    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-istio-token" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:46.684484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.674917    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-token" pod "fd993708-fadc-48af-99eb-ca7bde93dfe9" (UID: "fd993708-fadc-48af-99eb-ca7bde93dfe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:46.684484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.683767    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-envoy" pod "9ae1e29d-a609-43ca-b824-1851513008ac" (UID: "9ae1e29d-a609-43ca-b824-1851513008ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736619    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736653    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736668    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736683    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae1e29d-a609-43ca-b824-1851513008ac-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736698    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae1e29d-a609-43ca-b824-1851513008ac-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736711    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae1e29d-a609-43ca-b824-1851513008ac-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736724    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736738    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd993708-fadc-48af-99eb-ca7bde93dfe9-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.739593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:46.736751    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd993708-fadc-48af-99eb-ca7bde93dfe9-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:47.004632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.001689    5817 generic.go:356] "Generic (PLEG): container finished" podID="492cb988-75a8-4ca3-98a5-1d6584e88066" containerID="6c20a1487c323c797dabd921bbb85ce928e2e56a12e5839375938db7800a7e91" exitCode=0
Nov 03 10:52:47.004632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.001954    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp" event={"ID":"492cb988-75a8-4ca3-98a5-1d6584e88066","Type":"ContainerDied","Data":"6c20a1487c323c797dabd921bbb85ce928e2e56a12e5839375938db7800a7e91"}
Nov 03 10:52:47.022597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.022338    5817 generic.go:356] "Generic (PLEG): container finished" podID="9ae1e29d-a609-43ca-b824-1851513008ac" containerID="2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55" exitCode=0
Nov 03 10:52:47.022597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.022527    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"
Nov 03 10:52:47.023679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.023349    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" event={"ID":"9ae1e29d-a609-43ca-b824-1851513008ac","Type":"ContainerDied","Data":"2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55"}
Nov 03 10:52:47.023679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.023411    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl" event={"ID":"9ae1e29d-a609-43ca-b824-1851513008ac","Type":"ContainerDied","Data":"b93d9afeed4a9d071a0cfa7092b69031de6b6226ddaba144705c6d399f49d607"}
Nov 03 10:52:47.023679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.023438    5817 scope.go:122] "RemoveContainer" containerID="2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55"
Nov 03 10:52:47.025681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.025638    5817 generic.go:356] "Generic (PLEG): container finished" podID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerID="65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714" exitCode=0
Nov 03 10:52:47.025800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.025707    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" event={"ID":"fd993708-fadc-48af-99eb-ca7bde93dfe9","Type":"ContainerDied","Data":"65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714"}
Nov 03 10:52:47.025800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.025773    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd" event={"ID":"fd993708-fadc-48af-99eb-ca7bde93dfe9","Type":"ContainerDied","Data":"f72ca91f5a3939d0247e018d1122c769a817b198d348b9e1d02d89941ed67cd6"}
Nov 03 10:52:47.025954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.025936    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"
Nov 03 10:52:47.041767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.041662    5817 scope.go:122] "RemoveContainer" containerID="2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55"
Nov 03 10:52:47.042033 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:52:47.042004    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55\": container with ID starting with 2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55 not found: ID does not exist" containerID="2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55"
Nov 03 10:52:47.042135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.042047    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55"} err="failed to get container status \"2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55\": rpc error: code = NotFound desc = could not find container \"2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55\": container with ID starting with 2100aae09ff07996abf4c76bba6abd7272119929ff58cf11c3ee5a0cd3b23a55 not found: ID does not exist"
Nov 03 10:52:47.042135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.042068    5817 scope.go:122] "RemoveContainer" containerID="65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714"
Nov 03 10:52:47.053557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.053537    5817 scope.go:122] "RemoveContainer" containerID="65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714"
Nov 03 10:52:47.055178 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:52:47.055160    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714\": container with ID starting with 65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714 not found: ID does not exist" containerID="65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714"
Nov 03 10:52:47.055308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.055291    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714"} err="failed to get container status \"65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714\": rpc error: code = NotFound desc = could not find container \"65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714\": container with ID starting with 65697cf23d4d2f6221721be77083b1c855f0c8a42b5a8ced51a6e0db28026714 not found: ID does not exist"
Nov 03 10:52:47.057379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.057360    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"]
Nov 03 10:52:47.061425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.061405    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-4fkkl"]
Nov 03 10:52:47.070625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.070605    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"]
Nov 03 10:52:47.077405 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.077387    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-6j8vd"]
Nov 03 10:52:47.645537 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.642718    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:47.919807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.919723    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" path="/var/lib/kubelet/pods/9ae1e29d-a609-43ca-b824-1851513008ac/volumes"
Nov 03 10:52:47.920791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:47.920770    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" path="/var/lib/kubelet/pods/fd993708-fadc-48af-99eb-ca7bde93dfe9/volumes"
Nov 03 10:52:48.280856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.279643    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:48.334079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.334051    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:48.476033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.476005    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492cb988-75a8-4ca3-98a5-1d6584e88066-kube-api-access-q6d5n\" (UniqueName: \"kubernetes.io/projected/492cb988-75a8-4ca3-98a5-1d6584e88066-kube-api-access-q6d5n\") pod \"492cb988-75a8-4ca3-98a5-1d6584e88066\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") "
Nov 03 10:52:48.476212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.476072    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-xks-ca\") pod \"492cb988-75a8-4ca3-98a5-1d6584e88066\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") "
Nov 03 10:52:48.476212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.476099    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-openshift-ca\") pod \"492cb988-75a8-4ca3-98a5-1d6584e88066\" (UID: \"492cb988-75a8-4ca3-98a5-1d6584e88066\") "
Nov 03 10:52:48.476738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.476715    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-openshift-ca" pod "492cb988-75a8-4ca3-98a5-1d6584e88066" (UID: "492cb988-75a8-4ca3-98a5-1d6584e88066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:48.479052 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.479020    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-xks-ca" pod "492cb988-75a8-4ca3-98a5-1d6584e88066" (UID: "492cb988-75a8-4ca3-98a5-1d6584e88066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:52:48.479178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.479098    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492cb988-75a8-4ca3-98a5-1d6584e88066-kube-api-access-q6d5n" pod "492cb988-75a8-4ca3-98a5-1d6584e88066" (UID: "492cb988-75a8-4ca3-98a5-1d6584e88066"). InnerVolumeSpecName "kube-api-access-q6d5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:48.578330 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.578302    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6d5n\" (UniqueName: \"kubernetes.io/projected/492cb988-75a8-4ca3-98a5-1d6584e88066-kube-api-access-q6d5n\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:48.578479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.578333    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:48.578479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.578348    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492cb988-75a8-4ca3-98a5-1d6584e88066-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:48.972599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:48.967000    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:49.060424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:49.060403    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"
Nov 03 10:52:49.060704 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:49.060437    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp" event={"ID":"492cb988-75a8-4ca3-98a5-1d6584e88066","Type":"ContainerDied","Data":"aa7fff3370d3af2bf9f4bffee7f4ca151a97b3c983d73d4078d6adffb852d5b8"}
Nov 03 10:52:49.060789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:49.060732    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa7fff3370d3af2bf9f4bffee7f4ca151a97b3c983d73d4078d6adffb852d5b8"
Nov 03 10:52:49.563106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:49.563070    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:49.701086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:49.697403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:50.542605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:50.535955    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:51.182785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:51.181938    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:51.794597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:51.791602    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:52.051816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:52.051742    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:52.415672 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:52:52.415565    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:52:52.511463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:52.511433    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:52.870294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:52.868865    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": read tcp 10.133.0.2:51356->10.133.0.148:15021: read: connection reset by peer" start-of-body=
Nov 03 10:52:52.870294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:52.868926    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": read tcp 10.133.0.2:51356->10.133.0.148:15021: read: connection reset by peer"
Nov 03 10:52:53.098687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:53.098553    5817 generic.go:356] "Generic (PLEG): container finished" podID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerID="aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18" exitCode=0
Nov 03 10:52:53.098687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:53.098632    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" event={"ID":"c5173ad5-8182-4a4b-9bf0-a291aca5f363","Type":"ContainerDied","Data":"aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18"}
Nov 03 10:52:53.099163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:53.099145    5817 scope.go:122] "RemoveContainer" containerID="aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18"
Nov 03 10:52:53.101117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:53.101082    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:53.101193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:53.101171    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:53.101239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:53.101217    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:53.167939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:53.167886    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:53.757592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:53.754460    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:54.021586 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.021491    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"]
Nov 03 10:52:54.025955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.025926    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0956c4-pnvfx"]
Nov 03 10:52:54.123587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.123530    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" event={"ID":"c5173ad5-8182-4a4b-9bf0-a291aca5f363","Type":"ContainerStarted","Data":"3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85"}
Nov 03 10:52:54.382308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.378645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:54.469061 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.469027    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:54.469408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.469386    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 10:52:54.469478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.469455    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 10:52:54.821024 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.820989    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:54.959175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:54.959010    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:55.108654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:55.108234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:55.468996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:55.468962    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 10:52:55.469165 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:55.469023    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 10:52:55.927396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:55.927274    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427a43ca-e3f1-4616-be50-352afca82a2f" path="/var/lib/kubelet/pods/427a43ca-e3f1-4616-be50-352afca82a2f/volumes"
Nov 03 10:52:55.939631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:55.939603    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:56.468914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:56.468879    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 10:52:56.469131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:56.468951    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 10:52:56.559129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:56.559093    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:56.964686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:56.964653    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:57.107592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:57.097706    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:57.230051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:57.229923    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:57.437309 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:57.437251    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:57.468509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:57.468477    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 10:52:57.468735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:57.468541    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 10:52:58.073946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:58.073916    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:58.469347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:58.469306    5817 patch_prober.go:29] interesting pod/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 10:52:58.469546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:58.469365    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 10:52:58.816669 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:58.810373    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:52:59.473100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:59.473068    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:59.473535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:59.473479    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:59.474106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:59.474087    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:52:59.551792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:52:59.551747    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:00.263315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:00.263286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:00.848557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:00.848521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:01.180845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:01.180811    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:01.761445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:01.761413    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:01.935370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:01.935332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:02.631784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:02.631560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:03.295394 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:03.295350    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:03.439616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:03.439562    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:03.597667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:03.597514    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:03.639606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:03.639551    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"]
Nov 03 10:53:03.643970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:03.643935    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-835861-9klhp"]
Nov 03 10:53:03.905318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:03.905218    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492cb988-75a8-4ca3-98a5-1d6584e88066" path="/var/lib/kubelet/pods/492cb988-75a8-4ca3-98a5-1d6584e88066/volumes"
Nov 03 10:53:03.928102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:03.922171    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:04.155104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:04.155066    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:04.859978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:04.859946    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:05.403844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:05.403791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:05.621599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:05.617067    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:05.868126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:05.868094    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:06.036343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:06.036292    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:06.171611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:06.171510    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:06.404564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:06.404530    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:53:06.405475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:06.405452    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:06.896627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:06.896598    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:07.197534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:07.197497    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:07.552685 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:07.552405    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:08.133793 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:08.133755    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:08.283395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:08.283360    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:08.393911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:08.393825    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"]
Nov 03 10:53:08.394359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:08.394329    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy" containerID="cri-o://c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0" gracePeriod=30
Nov 03 10:53:08.428007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:08.427973    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:08.566050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:08.566013    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:09.206178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:09.206143    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:09.884240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:09.884207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:10.194564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:10.194528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:10.334785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:10.334749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:10.451454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:10.451365    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"]
Nov 03 10:53:10.452047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:10.452010    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy" containerID="cri-o://5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0" gracePeriod=30
Nov 03 10:53:10.542285 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:10.542080    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:10.891718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:10.891641    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:11.026657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.026627    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"]
Nov 03 10:53:11.027193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027179    5817 cpu_manager.go:414] "containerMap: removing container" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy"
Nov 03 10:53:11.027193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027193    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy"
Nov 03 10:53:11.027341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027232    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy"
Nov 03 10:53:11.027341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027237    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy"
Nov 03 10:53:11.027341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027251    5817 cpu_manager.go:414] "containerMap: removing container" podUID="492cb988-75a8-4ca3-98a5-1d6584e88066" containerName="revoke-keys"
Nov 03 10:53:11.027341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027257    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492cb988-75a8-4ca3-98a5-1d6584e88066" containerName="revoke-keys"
Nov 03 10:53:11.027553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027345    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae1e29d-a609-43ca-b824-1851513008ac" containerName="istio-proxy"
Nov 03 10:53:11.027553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027356    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="492cb988-75a8-4ca3-98a5-1d6584e88066" containerName="revoke-keys"
Nov 03 10:53:11.027553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.027364    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd993708-fadc-48af-99eb-ca7bde93dfe9" containerName="istio-proxy"
Nov 03 10:53:11.032342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.032320    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.036098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.036080    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:53:11.036839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.036821    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:53:11.038993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.038970    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"]
Nov 03 10:53:11.113442 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.113405    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.113643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.113586    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntv5v\" (UniqueName: \"kubernetes.io/projected/80584f3c-5b92-4dc6-ac90-c585db8e8b02-kube-api-access-ntv5v\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.113697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.113662    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.216723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.216685    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntv5v\" (UniqueName: \"kubernetes.io/projected/80584f3c-5b92-4dc6-ac90-c585db8e8b02-kube-api-access-ntv5v\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.216945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.216779    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.217054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.217031    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.217698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.217677    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.219332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.219300    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.224977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.224952    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntv5v\" (UniqueName: \"kubernetes.io/projected/80584f3c-5b92-4dc6-ac90-c585db8e8b02-kube-api-access-ntv5v\") pod \"maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.351356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.351325    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:11.636704 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.636671    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"]
Nov 03 10:53:11.965169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:11.965136    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:12.385061 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.385022    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw" event={"ID":"80584f3c-5b92-4dc6-ac90-c585db8e8b02","Type":"ContainerStarted","Data":"29dd3641a8388da92253de9a224605dc7ba76389ca3ebe1146bf9968ec926492"}
Nov 03 10:53:12.385429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.385078    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw" event={"ID":"80584f3c-5b92-4dc6-ac90-c585db8e8b02","Type":"ContainerStarted","Data":"b40dbebf5ed041a1a5c7672158ada4671667bcd2c227ebbf6718b8f5b787bb5b"}
Nov 03 10:53:12.403618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.403540    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw" podStartSLOduration=2.403526334 podStartE2EDuration="2.403526334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:12.402118504 +0000 UTC m=+1422.607700580" watchObservedRunningTime="2026-11-03 10:53:12.403526334 +0000 UTC m=+1422.609108381"
Nov 03 10:53:12.483744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.483712    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:12.485432 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.485407    5817 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:53:12.485541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.485462    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:53:12.625264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.625220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:12.751396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.750177    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"]
Nov 03 10:53:12.751396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.750988    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy" containerID="cri-o://600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99" gracePeriod=30
Nov 03 10:53:12.768753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.768320    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:12.915428 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:12.913964    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:13.396136 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.396103    5817 generic.go:356] "Generic (PLEG): container finished" podID="80584f3c-5b92-4dc6-ac90-c585db8e8b02" containerID="29dd3641a8388da92253de9a224605dc7ba76389ca3ebe1146bf9968ec926492" exitCode=0
Nov 03 10:53:13.396518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.396186    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw" event={"ID":"80584f3c-5b92-4dc6-ac90-c585db8e8b02","Type":"ContainerDied","Data":"29dd3641a8388da92253de9a224605dc7ba76389ca3ebe1146bf9968ec926492"}
Nov 03 10:53:13.703013 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.702949    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:53:13.779141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.779105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:13.884849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.884818    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-certs\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.884868    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-token\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-token\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.884916    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-kube-api-access-vq6qs\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-kube-api-access-vq6qs\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885013    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-socket\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885049    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-envoy\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885083    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-certs" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:13.885178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885104    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-data\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885162    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96596406-3fe4-45fb-add8-18425ca8f1e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96596406-3fe4-45fb-add8-18425ca8f1e7-istiod-ca-cert\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885197    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-credential-socket\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885252    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-podinfo\") pod \"96596406-3fe4-45fb-add8-18425ca8f1e7\" (UID: \"96596406-3fe4-45fb-add8-18425ca8f1e7\") "
Nov 03 10:53:13.885385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885323    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-socket" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:13.885385 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885342    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-data" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:13.885655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885535    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-credential-socket" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:13.885655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.885552    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96596406-3fe4-45fb-add8-18425ca8f1e7-istiod-ca-cert" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:13.886216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.886180    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96596406-3fe4-45fb-add8-18425ca8f1e7-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:13.886216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.886210    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:13.886360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.886225    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:13.886360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.886241    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:13.886360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.886258    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:13.887536 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.887516    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-podinfo" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:53:13.887771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.887746    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-kube-api-access-vq6qs" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "kube-api-access-vq6qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:13.887864 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.887807    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-token" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:13.887913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.887886    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-envoy" pod "96596406-3fe4-45fb-add8-18425ca8f1e7" (UID: "96596406-3fe4-45fb-add8-18425ca8f1e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:13.988151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.988075    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:13.988151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.988109    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:13.988151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.988125    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:13.988341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:13.988165    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6qs\" (UniqueName: \"kubernetes.io/projected/96596406-3fe4-45fb-add8-18425ca8f1e7-kube-api-access-vq6qs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.406760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.406671    5817 generic.go:356] "Generic (PLEG): container finished" podID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerID="c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0" exitCode=0
Nov 03 10:53:14.406760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.406751    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"
Nov 03 10:53:14.407281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.406754    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" event={"ID":"96596406-3fe4-45fb-add8-18425ca8f1e7","Type":"ContainerDied","Data":"c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0"}
Nov 03 10:53:14.407281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.406876    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t" event={"ID":"96596406-3fe4-45fb-add8-18425ca8f1e7","Type":"ContainerDied","Data":"db1be2ca36343c5c747cd6605a3fece52e6481b8639fdca27fc2f6c532e02006"}
Nov 03 10:53:14.407281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.406899    5817 scope.go:122] "RemoveContainer" containerID="c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0"
Nov 03 10:53:14.418941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.418920    5817 scope.go:122] "RemoveContainer" containerID="c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0"
Nov 03 10:53:14.419206 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:14.419188    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0\": container with ID starting with c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0 not found: ID does not exist" containerID="c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0"
Nov 03 10:53:14.419291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.419216    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0"} err="failed to get container status \"c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0\": rpc error: code = NotFound desc = could not find container \"c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0\": container with ID starting with c8b49f3ad5456e65c2743c598a680f45eac601d6e19d97fa87bd526af387a7d0 not found: ID does not exist"
Nov 03 10:53:14.435684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.435651    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"]
Nov 03 10:53:14.439268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.439243    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-9w44t"]
Nov 03 10:53:14.463630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.463589    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:14.610759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.610724    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:14.801454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.801420    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-xks-ca\") pod \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") "
Nov 03 10:53:14.801676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.801471    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80584f3c-5b92-4dc6-ac90-c585db8e8b02-kube-api-access-ntv5v\" (UniqueName: \"kubernetes.io/projected/80584f3c-5b92-4dc6-ac90-c585db8e8b02-kube-api-access-ntv5v\") pod \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") "
Nov 03 10:53:14.801880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.801859    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-openshift-ca\") pod \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\" (UID: \"80584f3c-5b92-4dc6-ac90-c585db8e8b02\") "
Nov 03 10:53:14.802230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.802206    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-openshift-ca" pod "80584f3c-5b92-4dc6-ac90-c585db8e8b02" (UID: "80584f3c-5b92-4dc6-ac90-c585db8e8b02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:14.802606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.802456    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.803821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.803795    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-xks-ca" pod "80584f3c-5b92-4dc6-ac90-c585db8e8b02" (UID: "80584f3c-5b92-4dc6-ac90-c585db8e8b02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:14.804180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.804158    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80584f3c-5b92-4dc6-ac90-c585db8e8b02-kube-api-access-ntv5v" pod "80584f3c-5b92-4dc6-ac90-c585db8e8b02" (UID: "80584f3c-5b92-4dc6-ac90-c585db8e8b02"). InnerVolumeSpecName "kube-api-access-ntv5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:14.904311 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.904278    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80584f3c-5b92-4dc6-ac90-c585db8e8b02-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.904311 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:14.904313    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntv5v\" (UniqueName: \"kubernetes.io/projected/80584f3c-5b92-4dc6-ac90-c585db8e8b02-kube-api-access-ntv5v\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.255601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.255459    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:15.416392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.416361    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"
Nov 03 10:53:15.416392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.416381    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw" event={"ID":"80584f3c-5b92-4dc6-ac90-c585db8e8b02","Type":"ContainerDied","Data":"b40dbebf5ed041a1a5c7672158ada4671667bcd2c227ebbf6718b8f5b787bb5b"}
Nov 03 10:53:15.416940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.416430    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b40dbebf5ed041a1a5c7672158ada4671667bcd2c227ebbf6718b8f5b787bb5b"
Nov 03 10:53:15.773267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.773246    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822039    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-data\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822091    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istiod-ca-cert\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822170    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-kube-api-access-gmwt7\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-kube-api-access-gmwt7\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822219    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-certs\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822265    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-token\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-token\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822406    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-envoy\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822489    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-credential-socket\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822534    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-socket\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.822679    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-podinfo\") pod \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\" (UID: \"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f\") "
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.824965    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"]
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.825923    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.825943    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.825961    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.825968    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.825982    5817 cpu_manager.go:414] "containerMap: removing container" podUID="80584f3c-5b92-4dc6-ac90-c585db8e8b02" containerName="revoke-keys"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.825989    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80584f3c-5b92-4dc6-ac90-c585db8e8b02" containerName="revoke-keys"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.826018    5817 cpu_manager.go:414] "containerMap: removing container" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.826024    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.826181    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.826195    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerName="istio-proxy"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.826206    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="80584f3c-5b92-4dc6-ac90-c585db8e8b02" containerName="revoke-keys"
Nov 03 10:53:15.829680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.826221    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" containerName="istio-proxy"
Nov 03 10:53:15.830970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.830041    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-podinfo" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:53:15.830970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.830290    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-data" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:15.830970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.830562    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istiod-ca-cert" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:15.877931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.877887    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:15.878242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.878207    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-credential-socket" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:15.878465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.878440    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-socket" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:15.878587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.878483    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-certs" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:15.881882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.881803    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:53:15.882185 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.882150    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:53:15.886199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.886170    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-envoy" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:15.887836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.886682    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-token" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:15.890256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.890205    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"]
Nov 03 10:53:15.891531 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.891495    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-kube-api-access-gmwt7" pod "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" (UID: "e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f"). InnerVolumeSpecName "kube-api-access-gmwt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:15.910722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.910694    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96596406-3fe4-45fb-add8-18425ca8f1e7" path="/var/lib/kubelet/pods/96596406-3fe4-45fb-add8-18425ca8f1e7/volumes"
Nov 03 10:53:15.928596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928243    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:15.928596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928363    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpcfg\" (UniqueName: \"kubernetes.io/projected/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-kube-api-access-vpcfg\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:15.928596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928421    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928926    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928949    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928961    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928973    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928985    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.928997    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmwt7\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-kube-api-access-gmwt7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.929011    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.929022    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:15.929108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:15.929034    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:16.031981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.030026    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:16.031981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.030161    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpcfg\" (UniqueName: \"kubernetes.io/projected/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-kube-api-access-vpcfg\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:16.031981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.030221    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:16.031981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.031395    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:16.039269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.039245    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:16.044735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.044705    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpcfg\" (UniqueName: \"kubernetes.io/projected/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-kube-api-access-vpcfg\") pod \"maas-api-revoke-sub-04e41c7c-npm6h\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:16.146615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.146485    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:16.240529 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.240498    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:16.430897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.430862    5817 generic.go:356] "Generic (PLEG): container finished" podID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" containerID="5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0" exitCode=0
Nov 03 10:53:16.431338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.430921    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"
Nov 03 10:53:16.431338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.430943    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" event={"ID":"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f","Type":"ContainerDied","Data":"5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0"}
Nov 03 10:53:16.431338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.431009    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7" event={"ID":"e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f","Type":"ContainerDied","Data":"4b511259c16d718dd2c67a7c1e37c31ee125c5b58a37bc169e4e9d67c472a491"}
Nov 03 10:53:16.431338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.431039    5817 scope.go:122] "RemoveContainer" containerID="5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0"
Nov 03 10:53:16.445083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.445063    5817 scope.go:122] "RemoveContainer" containerID="8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376"
Nov 03 10:53:16.454719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.454703    5817 scope.go:122] "RemoveContainer" containerID="5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0"
Nov 03 10:53:16.455011 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:16.454993    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0\": container with ID starting with 5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0 not found: ID does not exist" containerID="5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0"
Nov 03 10:53:16.455059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.455020    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0"} err="failed to get container status \"5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0\": rpc error: code = NotFound desc = could not find container \"5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0\": container with ID starting with 5bdda9ba3a0c9e1f1418a7976ea9e3b3fc2996345ea756ec3c7e718bbcb2f5d0 not found: ID does not exist"
Nov 03 10:53:16.455059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.455039    5817 scope.go:122] "RemoveContainer" containerID="8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376"
Nov 03 10:53:16.455265 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:16.455247    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376\": container with ID starting with 8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376 not found: ID does not exist" containerID="8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376"
Nov 03 10:53:16.455321 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.455274    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376"} err="failed to get container status \"8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376\": rpc error: code = NotFound desc = could not find container \"8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376\": container with ID starting with 8e57a4825734e23249a43fa89056234b425a30d81e42822030abd2d27c759376 not found: ID does not exist"
Nov 03 10:53:16.462665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.462638    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"]
Nov 03 10:53:16.475632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.467200    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-835861-openshift-default-54d5597d4f-958p7"]
Nov 03 10:53:16.476185 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.476153    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:16.529733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:16.529705    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"]
Nov 03 10:53:16.530100 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:53:16.530066    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffebe9ce_8cac_48b8_b3bb_fd6d75ead2c9.slice/crio-dbfcc16b5df659ce031a1d0e8a996a7687260fc2015c7f61ef9c3d0b83b6c901 WatchSource:0}: Error finding container dbfcc16b5df659ce031a1d0e8a996a7687260fc2015c7f61ef9c3d0b83b6c901: Status 404 returned error can't find the container with id dbfcc16b5df659ce031a1d0e8a996a7687260fc2015c7f61ef9c3d0b83b6c901
Nov 03 10:53:17.064647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:17.056726    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:17.445213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:17.445177    5817 generic.go:356] "Generic (PLEG): container finished" podID="ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9" containerID="2d71f598bc76c5c06b1b2c385a8e839c63d4346fe3c76c0b9bc38a58fe1d2146" exitCode=0
Nov 03 10:53:17.445755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:17.445256    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h" event={"ID":"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9","Type":"ContainerDied","Data":"2d71f598bc76c5c06b1b2c385a8e839c63d4346fe3c76c0b9bc38a58fe1d2146"}
Nov 03 10:53:17.445755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:17.445308    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h" event={"ID":"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9","Type":"ContainerStarted","Data":"dbfcc16b5df659ce031a1d0e8a996a7687260fc2015c7f61ef9c3d0b83b6c901"}
Nov 03 10:53:17.710599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:17.706484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:17.905464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:17.905431    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f" path="/var/lib/kubelet/pods/e5c59bd2-36ad-4d6b-a4b3-0bcbe488095f/volumes"
Nov 03 10:53:18.046648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.046625    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:53:18.170167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170127    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-podinfo\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.170313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170186    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-credential-socket\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.170313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170216    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-certs\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.170313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170253    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-socket\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.170478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170344    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-envoy\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.170478 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170462    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-token\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-token\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.170605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170499    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-kube-api-access-6hzjz\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-kube-api-access-6hzjz\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.170605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170508    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-certs" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:18.170605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170524    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-credential-socket" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:18.170775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170622    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istiod-ca-cert\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.170775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170681    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-socket" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:18.170775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.170745    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-data\") pod \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\" (UID: \"d36b771b-d757-4cad-9ebf-12b74ed74bbe\") "
Nov 03 10:53:18.171187 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.171164    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-data" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:18.171539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.171482    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.171539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.171517    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.171539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.171532    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.171817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.171282    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istiod-ca-cert" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:18.172771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.172745    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-envoy" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:18.172945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.172923    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-podinfo" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:53:18.173347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.173324    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-token" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:18.173410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.173344    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-kube-api-access-6hzjz" pod "d36b771b-d757-4cad-9ebf-12b74ed74bbe" (UID: "d36b771b-d757-4cad-9ebf-12b74ed74bbe"). InnerVolumeSpecName "kube-api-access-6hzjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:18.272719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.272653    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.272719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.272679    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.272719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.272688    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.272719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.272697    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.272719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.272705    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzjz\" (UniqueName: \"kubernetes.io/projected/d36b771b-d757-4cad-9ebf-12b74ed74bbe-kube-api-access-6hzjz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.272719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.272714    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b771b-d757-4cad-9ebf-12b74ed74bbe-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.333710 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.332337    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:18.457359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.457309    5817 generic.go:356] "Generic (PLEG): container finished" podID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerID="600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99" exitCode=0
Nov 03 10:53:18.457932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.457373    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"
Nov 03 10:53:18.457932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.457469    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" event={"ID":"d36b771b-d757-4cad-9ebf-12b74ed74bbe","Type":"ContainerDied","Data":"600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99"}
Nov 03 10:53:18.457932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.457520    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb" event={"ID":"d36b771b-d757-4cad-9ebf-12b74ed74bbe","Type":"ContainerDied","Data":"80ace8d18dcabd3188dd314d25182de851ddd5b4e339664bd1ca80445bc24bba"}
Nov 03 10:53:18.457932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.457545    5817 scope.go:122] "RemoveContainer" containerID="600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99"
Nov 03 10:53:18.470857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.470840    5817 scope.go:122] "RemoveContainer" containerID="70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf"
Nov 03 10:53:18.485352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.485303    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"]
Nov 03 10:53:18.485972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.485951    5817 scope.go:122] "RemoveContainer" containerID="600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99"
Nov 03 10:53:18.486320 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:18.486291    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99\": container with ID starting with 600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99 not found: ID does not exist" containerID="600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99"
Nov 03 10:53:18.486404 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.486342    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99"} err="failed to get container status \"600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99\": rpc error: code = NotFound desc = could not find container \"600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99\": container with ID starting with 600298f9a87c7201e97074e818bfe11562185a81577c7fc87a3862d9f7c5ae99 not found: ID does not exist"
Nov 03 10:53:18.486404 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.486363    5817 scope.go:122] "RemoveContainer" containerID="70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf"
Nov 03 10:53:18.486644 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:18.486620    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf\": container with ID starting with 70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf not found: ID does not exist" containerID="70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf"
Nov 03 10:53:18.486644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.486653    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf"} err="failed to get container status \"70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf\": rpc error: code = NotFound desc = could not find container \"70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf\": container with ID starting with 70112e0de39c555a92c0ee754f746d3641cb8c6c022c6a0a2d351e72018578cf not found: ID does not exist"
Nov 03 10:53:18.490489 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.489402    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f81a79-openshift-default-5669fdddb5-rcqpb"]
Nov 03 10:53:18.609811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.609783    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:18.679071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.679040    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-openshift-ca\") pod \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") "
Nov 03 10:53:18.679255 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.679137    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-kube-api-access-vpcfg\" (UniqueName: \"kubernetes.io/projected/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-kube-api-access-vpcfg\") pod \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") "
Nov 03 10:53:18.679329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.679280    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-xks-ca\") pod \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\" (UID: \"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9\") "
Nov 03 10:53:18.679474 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.679448    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-openshift-ca" pod "ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9" (UID: "ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:18.679786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.679747    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.681443 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.681421    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-xks-ca" pod "ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9" (UID: "ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:18.681602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.681554    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-kube-api-access-vpcfg" pod "ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9" (UID: "ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9"). InnerVolumeSpecName "kube-api-access-vpcfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:18.781729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.781700    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:18.781858 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:18.781730    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpcfg\" (UniqueName: \"kubernetes.io/projected/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9-kube-api-access-vpcfg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:19.041123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:19.041087    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:19.467445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:19.467406    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h" event={"ID":"ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9","Type":"ContainerDied","Data":"dbfcc16b5df659ce031a1d0e8a996a7687260fc2015c7f61ef9c3d0b83b6c901"}
Nov 03 10:53:19.467445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:19.467436    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"
Nov 03 10:53:19.467900 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:19.467461    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbfcc16b5df659ce031a1d0e8a996a7687260fc2015c7f61ef9c3d0b83b6c901"
Nov 03 10:53:19.751813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:19.751658    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:19.903120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:19.903090    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" path="/var/lib/kubelet/pods/d36b771b-d757-4cad-9ebf-12b74ed74bbe/volumes"
Nov 03 10:53:20.473463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:20.473429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:21.253937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:21.253909    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:21.385425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:21.385390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:22.126141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:22.126106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:22.414764 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:22.414681    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:53:22.937230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:22.935082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:23.457284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:23.457249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:23.698275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:23.698244    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:23.928733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:23.928696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:23.929035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:23.928717    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"]
Nov 03 10:53:23.932738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:23.932714    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-08c392-bwxsw"]
Nov 03 10:53:24.076816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.076774    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:24.085214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.085187    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"]
Nov 03 10:53:24.085819 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.085804    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy"
Nov 03 10:53:24.085897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.085820    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy"
Nov 03 10:53:24.085897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.085834    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy"
Nov 03 10:53:24.085897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.085839    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy"
Nov 03 10:53:24.085897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.085856    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9" containerName="revoke-keys"
Nov 03 10:53:24.085897 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.085861    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9" containerName="revoke-keys"
Nov 03 10:53:24.086104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.086089    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy"
Nov 03 10:53:24.086139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.086109    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36b771b-d757-4cad-9ebf-12b74ed74bbe" containerName="istio-proxy"
Nov 03 10:53:24.086139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.086126    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9" containerName="revoke-keys"
Nov 03 10:53:24.091645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.091559    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.106624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.099081    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:53:24.106624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.104717    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:53:24.108414 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.108389    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"]
Nov 03 10:53:24.158815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.158781    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.158977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.158925    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcp2k\" (UniqueName: \"kubernetes.io/projected/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-kube-api-access-wcp2k\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.159064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.159047    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.210852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.210766    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:24.260858 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.260832    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.261085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.260902    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcp2k\" (UniqueName: \"kubernetes.io/projected/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-kube-api-access-wcp2k\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.261085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.261036    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.261754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.261679    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.263335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.263314    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.269076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.269056    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcp2k\" (UniqueName: \"kubernetes.io/projected/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-kube-api-access-wcp2k\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.421439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.421405    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:24.517386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.515975    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:24.700239 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.697507    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:24.769774 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.763435    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"]
Nov 03 10:53:24.772055 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:53:24.772026    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb5855e_69f6_47ad_9ef9_135bd1c8961c.slice/crio-0caf9c0fe16e93256daad1daaf098ebd266015c07521ddcc14e8ffef993f67e1 WatchSource:0}: Error finding container 0caf9c0fe16e93256daad1daaf098ebd266015c07521ddcc14e8ffef993f67e1: Status 404 returned error can't find the container with id 0caf9c0fe16e93256daad1daaf098ebd266015c07521ddcc14e8ffef993f67e1
Nov 03 10:53:24.995077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:24.995039    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:25.403054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:25.402900    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:25.534456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:25.534416    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5" event={"ID":"8bb5855e-69f6-47ad-9ef9-135bd1c8961c","Type":"ContainerStarted","Data":"3a8812eb621fad33c008084e6dceb88529d6fc68adbb75e4b3e261a3933671df"}
Nov 03 10:53:25.534934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:25.534514    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5" event={"ID":"8bb5855e-69f6-47ad-9ef9-135bd1c8961c","Type":"ContainerStarted","Data":"0caf9c0fe16e93256daad1daaf098ebd266015c07521ddcc14e8ffef993f67e1"}
Nov 03 10:53:25.552426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:25.552368    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5" podStartSLOduration=1.552349049 podStartE2EDuration="1.552349049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:25.549514343 +0000 UTC m=+1435.755096395" watchObservedRunningTime="2026-11-03 10:53:25.552349049 +0000 UTC m=+1435.757931105"
Nov 03 10:53:25.904156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:25.904101    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80584f3c-5b92-4dc6-ac90-c585db8e8b02" path="/var/lib/kubelet/pods/80584f3c-5b92-4dc6-ac90-c585db8e8b02/volumes"
Nov 03 10:53:26.176642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:26.176521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:26.544609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:26.544550    5817 generic.go:356] "Generic (PLEG): container finished" podID="8bb5855e-69f6-47ad-9ef9-135bd1c8961c" containerID="3a8812eb621fad33c008084e6dceb88529d6fc68adbb75e4b3e261a3933671df" exitCode=0
Nov 03 10:53:26.544973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:26.544610    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5" event={"ID":"8bb5855e-69f6-47ad-9ef9-135bd1c8961c","Type":"ContainerDied","Data":"3a8812eb621fad33c008084e6dceb88529d6fc68adbb75e4b3e261a3933671df"}
Nov 03 10:53:26.964924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:26.964882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:27.653705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.653417    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:27.874413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.874379    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:27.954787 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.954758    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-xks-ca\") pod \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") "
Nov 03 10:53:27.954951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.954836    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-openshift-ca\") pod \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") "
Nov 03 10:53:27.954951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.954934    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-kube-api-access-wcp2k\" (UniqueName: \"kubernetes.io/projected/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-kube-api-access-wcp2k\") pod \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\" (UID: \"8bb5855e-69f6-47ad-9ef9-135bd1c8961c\") "
Nov 03 10:53:27.955175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.955149    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-openshift-ca" pod "8bb5855e-69f6-47ad-9ef9-135bd1c8961c" (UID: "8bb5855e-69f6-47ad-9ef9-135bd1c8961c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:27.955445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.955432    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:27.957107 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.957084    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-xks-ca" pod "8bb5855e-69f6-47ad-9ef9-135bd1c8961c" (UID: "8bb5855e-69f6-47ad-9ef9-135bd1c8961c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:27.957200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:27.957181    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-kube-api-access-wcp2k" pod "8bb5855e-69f6-47ad-9ef9-135bd1c8961c" (UID: "8bb5855e-69f6-47ad-9ef9-135bd1c8961c"). InnerVolumeSpecName "kube-api-access-wcp2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:28.057083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:28.057049    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcp2k\" (UniqueName: \"kubernetes.io/projected/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-kube-api-access-wcp2k\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:28.057083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:28.057084    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb5855e-69f6-47ad-9ef9-135bd1c8961c-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:28.476151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:28.476114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:28.572614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:28.572586    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"
Nov 03 10:53:28.572824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:28.572587    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5" event={"ID":"8bb5855e-69f6-47ad-9ef9-135bd1c8961c","Type":"ContainerDied","Data":"0caf9c0fe16e93256daad1daaf098ebd266015c07521ddcc14e8ffef993f67e1"}
Nov 03 10:53:28.572824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:28.572724    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0caf9c0fe16e93256daad1daaf098ebd266015c07521ddcc14e8ffef993f67e1"
Nov 03 10:53:28.863105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:28.863014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:30.627099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:30.627051    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:30.791209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:30.791173    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"]
Nov 03 10:53:30.791713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:30.791661    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy" containerID="cri-o://3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85" gracePeriod=30
Nov 03 10:53:30.858935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:30.858897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:31.287649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:31.287619    5817 scope.go:122] "RemoveContainer" containerID="fe531d559d786c6dde61d4b700564bd300af979c98ba410081443fce094f9cbf"
Nov 03 10:53:31.298495 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:31.298477    5817 scope.go:122] "RemoveContainer" containerID="c0e52f2b712ce87a6089cca429d91ee71ea08d9d7f799353bb5509c766935c1a"
Nov 03 10:53:31.308534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:31.308513    5817 scope.go:122] "RemoveContainer" containerID="f60209036d2b2770ac0e04affb68e3489b021f35192fcbbd2517bd0997911875"
Nov 03 10:53:31.562944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:31.562865    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:32.119351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:32.119320    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:32.248718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:32.248684    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:32.388103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:32.388015    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:33.059027 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:33.058990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:33.899638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:33.899600    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:34.563681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:34.561107    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:35.193693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:35.193660    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:35.841914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:35.841886    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:35.968110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:35.968083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:36.095968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.095942    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:53:36.100858 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.100818    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:36.178905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.178876    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-credential-socket\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.178923    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-kube-api-access-qvf5l\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-kube-api-access-qvf5l\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.178955    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-socket\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179051 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.178994    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istiod-ca-cert\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179052    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-data\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179109    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-token\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-token\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179152    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-envoy\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179183    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-credential-socket" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:36.179332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179260    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-podinfo\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179302    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-socket" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:36.179491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179347    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-data" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:36.179491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179363    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-certs\") pod \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\" (UID: \"c5173ad5-8182-4a4b-9bf0-a291aca5f363\") "
Nov 03 10:53:36.179491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179457    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istiod-ca-cert" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:36.179688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179672    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-certs" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:36.179918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179861    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.179918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179879    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.179918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179888    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.179918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179896    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.179918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.179906    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.181456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.181432    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-podinfo" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:53:36.181613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.181561    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-token" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:36.181823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.181805    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-envoy" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:36.181965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.181943    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-kube-api-access-qvf5l" pod "c5173ad5-8182-4a4b-9bf0-a291aca5f363" (UID: "c5173ad5-8182-4a4b-9bf0-a291aca5f363"). InnerVolumeSpecName "kube-api-access-qvf5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:36.280875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.280847    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.280875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.280875    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.281359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.280889    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5173ad5-8182-4a4b-9bf0-a291aca5f363-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.281359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.280902    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvf5l\" (UniqueName: \"kubernetes.io/projected/c5173ad5-8182-4a4b-9bf0-a291aca5f363-kube-api-access-qvf5l\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:36.540884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.540851    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:36.644795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.644759    5817 generic.go:356] "Generic (PLEG): container finished" podID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerID="3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85" exitCode=0
Nov 03 10:53:36.644958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.644809    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"
Nov 03 10:53:36.644958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.644841    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" event={"ID":"c5173ad5-8182-4a4b-9bf0-a291aca5f363","Type":"ContainerDied","Data":"3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85"}
Nov 03 10:53:36.644958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.644892    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh" event={"ID":"c5173ad5-8182-4a4b-9bf0-a291aca5f363","Type":"ContainerDied","Data":"bcf68a3fb1b9765057b67b0e67d5aca842baa94e4f531ef2962558490abfc82e"}
Nov 03 10:53:36.644958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.644916    5817 scope.go:122] "RemoveContainer" containerID="3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85"
Nov 03 10:53:36.657789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.657772    5817 scope.go:122] "RemoveContainer" containerID="aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18"
Nov 03 10:53:36.667471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.667207    5817 scope.go:122] "RemoveContainer" containerID="3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85"
Nov 03 10:53:36.669599 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:36.668268    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85\": container with ID starting with 3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85 not found: ID does not exist" containerID="3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85"
Nov 03 10:53:36.669599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.668306    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85"} err="failed to get container status \"3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85\": rpc error: code = NotFound desc = could not find container \"3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85\": container with ID starting with 3b360477e909547f701c4a567e963e9ccecba47fd07e026b59cf01910fe5cd85 not found: ID does not exist"
Nov 03 10:53:36.669599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.668331    5817 scope.go:122] "RemoveContainer" containerID="aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18"
Nov 03 10:53:36.669599 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:36.668649    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18\": container with ID starting with aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18 not found: ID does not exist" containerID="aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18"
Nov 03 10:53:36.669599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.668671    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18"} err="failed to get container status \"aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18\": rpc error: code = NotFound desc = could not find container \"aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18\": container with ID starting with aa9fb28adecfca47cf54c5ed9745a2999b395f04f44546d0bcca4b4ba8212a18 not found: ID does not exist"
Nov 03 10:53:36.672828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.672806    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"]
Nov 03 10:53:36.675382 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.675361    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:36.677071 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.677055    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-08c392-openshift-default-768c847c6f-tczrh"]
Nov 03 10:53:36.816647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:36.814965    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:37.022409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:37.022358    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:37.028053 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:37.028025    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"]
Nov 03 10:53:37.032281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:37.032253    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa420c-58nf5"]
Nov 03 10:53:37.161542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:37.161461    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:37.595170 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:37.595138    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:37.904284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:37.904204    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb5855e-69f6-47ad-9ef9-135bd1c8961c" path="/var/lib/kubelet/pods/8bb5855e-69f6-47ad-9ef9-135bd1c8961c/volumes"
Nov 03 10:53:37.904680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:37.904664    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" path="/var/lib/kubelet/pods/c5173ad5-8182-4a4b-9bf0-a291aca5f363/volumes"
Nov 03 10:53:37.961919 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:37.961358    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:38.357767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.357734    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"]
Nov 03 10:53:38.358376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358354    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy"
Nov 03 10:53:38.358487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358378    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy"
Nov 03 10:53:38.358487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358400    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy"
Nov 03 10:53:38.358487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358407    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy"
Nov 03 10:53:38.358487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358462    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8bb5855e-69f6-47ad-9ef9-135bd1c8961c" containerName="revoke-keys"
Nov 03 10:53:38.358487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358470    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb5855e-69f6-47ad-9ef9-135bd1c8961c" containerName="revoke-keys"
Nov 03 10:53:38.358873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358634    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy"
Nov 03 10:53:38.358873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358652    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5173ad5-8182-4a4b-9bf0-a291aca5f363" containerName="istio-proxy"
Nov 03 10:53:38.358873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.358676    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb5855e-69f6-47ad-9ef9-135bd1c8961c" containerName="revoke-keys"
Nov 03 10:53:38.363708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.363689    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.369755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.369731    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"]
Nov 03 10:53:38.445174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.445143    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:53:38.445334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.445185    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:53:38.544264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.544237    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.544410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.544282    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd88c\" (UniqueName: \"kubernetes.io/projected/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-kube-api-access-dd88c\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.544485 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.544417    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.645987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.645909    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.645987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.645960    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd88c\" (UniqueName: \"kubernetes.io/projected/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-kube-api-access-dd88c\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.646440 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.646032    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.646493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.646474    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.648782 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.648756    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.656993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.656971    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd88c\" (UniqueName: \"kubernetes.io/projected/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-kube-api-access-dd88c\") pod \"maas-api-revoke-sub-f4e3b305-q2d6b\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.679779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.679760    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:38.776741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.776560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:38.969915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:38.969883    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"]
Nov 03 10:53:38.970417 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:53:38.970390    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf092b867_be96_4a2c_b01f_9dbb50e3e7cc.slice/crio-1aed2f31af2400958e6764c5fae8679172920ff8d68067ecbb11c872ec53b9ca WatchSource:0}: Error finding container 1aed2f31af2400958e6764c5fae8679172920ff8d68067ecbb11c872ec53b9ca: Status 404 returned error can't find the container with id 1aed2f31af2400958e6764c5fae8679172920ff8d68067ecbb11c872ec53b9ca
Nov 03 10:53:39.465835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:39.465796    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:39.670899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:39.670824    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b" event={"ID":"f092b867-be96-4a2c-b01f-9dbb50e3e7cc","Type":"ContainerStarted","Data":"9467954b4570ca30229c783457171387889d89c6d6a90bc3dc62cd60fa9e33a1"}
Nov 03 10:53:39.670899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:39.670871    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b" event={"ID":"f092b867-be96-4a2c-b01f-9dbb50e3e7cc","Type":"ContainerStarted","Data":"1aed2f31af2400958e6764c5fae8679172920ff8d68067ecbb11c872ec53b9ca"}
Nov 03 10:53:39.689029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:39.688977    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b" podStartSLOduration=1.6889652320000001 podStartE2EDuration="1.688965232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:39.685110016 +0000 UTC m=+1449.890692068" watchObservedRunningTime="2026-11-03 10:53:39.688965232 +0000 UTC m=+1449.894547277"
Nov 03 10:53:40.139045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:40.139011    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:40.296995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:40.296962    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:53:40.682865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:40.682832    5817 generic.go:356] "Generic (PLEG): container finished" podID="f092b867-be96-4a2c-b01f-9dbb50e3e7cc" containerID="9467954b4570ca30229c783457171387889d89c6d6a90bc3dc62cd60fa9e33a1" exitCode=0
Nov 03 10:53:40.683291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:40.682913    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b" event={"ID":"f092b867-be96-4a2c-b01f-9dbb50e3e7cc","Type":"ContainerDied","Data":"9467954b4570ca30229c783457171387889d89c6d6a90bc3dc62cd60fa9e33a1"}
Nov 03 10:53:41.644283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:41.644248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:42.084514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.084493    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:42.086259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.086234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:42.180052 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.180015    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-kube-api-access-dd88c\" (UniqueName: \"kubernetes.io/projected/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-kube-api-access-dd88c\") pod \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") "
Nov 03 10:53:42.180208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.180111    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-xks-ca\") pod \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") "
Nov 03 10:53:42.180291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.180221    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-openshift-ca\") pod \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\" (UID: \"f092b867-be96-4a2c-b01f-9dbb50e3e7cc\") "
Nov 03 10:53:42.182905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.182871    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-openshift-ca" pod "f092b867-be96-4a2c-b01f-9dbb50e3e7cc" (UID: "f092b867-be96-4a2c-b01f-9dbb50e3e7cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:42.183498 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.183466    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-xks-ca" pod "f092b867-be96-4a2c-b01f-9dbb50e3e7cc" (UID: "f092b867-be96-4a2c-b01f-9dbb50e3e7cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:42.184999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.184975    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-kube-api-access-dd88c" pod "f092b867-be96-4a2c-b01f-9dbb50e3e7cc" (UID: "f092b867-be96-4a2c-b01f-9dbb50e3e7cc"). InnerVolumeSpecName "kube-api-access-dd88c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:42.281719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.281689    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:42.281719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.281717    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd88c\" (UniqueName: \"kubernetes.io/projected/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-kube-api-access-dd88c\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:42.281719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.281726    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f092b867-be96-4a2c-b01f-9dbb50e3e7cc-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:42.646597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.643528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:42.768908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.768878    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"
Nov 03 10:53:42.769096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.768911    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b" event={"ID":"f092b867-be96-4a2c-b01f-9dbb50e3e7cc","Type":"ContainerDied","Data":"1aed2f31af2400958e6764c5fae8679172920ff8d68067ecbb11c872ec53b9ca"}
Nov 03 10:53:42.769096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:42.768972    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aed2f31af2400958e6764c5fae8679172920ff8d68067ecbb11c872ec53b9ca"
Nov 03 10:53:43.533264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:43.533209    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:44.137846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.136848    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:44.252029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.251963    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"]
Nov 03 10:53:44.252539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.252525    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f092b867-be96-4a2c-b01f-9dbb50e3e7cc" containerName="revoke-keys"
Nov 03 10:53:44.252609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.252540    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f092b867-be96-4a2c-b01f-9dbb50e3e7cc" containerName="revoke-keys"
Nov 03 10:53:44.252705 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.252695    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f092b867-be96-4a2c-b01f-9dbb50e3e7cc" containerName="revoke-keys"
Nov 03 10:53:44.256243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.256224    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.263987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.263955    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:53:44.264127 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.264062    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:53:44.265421 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.265148    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"]
Nov 03 10:53:44.290208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.287956    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:44.318215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.318183    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2dw8\" (UniqueName: \"kubernetes.io/projected/d8c160ce-323b-477a-8dbf-dbab771c50c9-kube-api-access-w2dw8\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.318408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.318263    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.318534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.318507    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.419809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.419714    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.419954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.419899    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2dw8\" (UniqueName: \"kubernetes.io/projected/d8c160ce-323b-477a-8dbf-dbab771c50c9-kube-api-access-w2dw8\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.420038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.419957    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.420548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.420513    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.423099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.423070    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.432456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.432433    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2dw8\" (UniqueName: \"kubernetes.io/projected/d8c160ce-323b-477a-8dbf-dbab771c50c9-kube-api-access-w2dw8\") pod \"maas-api-revoke-sub-c2cfe356-lqdk6\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.453269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.453243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:44.578733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.578704    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:44.588715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.588290    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:44.904186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:44.904152    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"]
Nov 03 10:53:45.267302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:45.267268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:45.811064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:45.811028    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6" event={"ID":"d8c160ce-323b-477a-8dbf-dbab771c50c9","Type":"ContainerStarted","Data":"6e6b3d8c938fbfe79968a77fdb591095f130844f4c953de0c9ed649c44e7318c"}
Nov 03 10:53:45.811433 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:45.811093    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6" event={"ID":"d8c160ce-323b-477a-8dbf-dbab771c50c9","Type":"ContainerStarted","Data":"d75626945be377e69891de6505d894da74e5938771cc53a9d1d94ca015fc2738"}
Nov 03 10:53:45.828022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:45.827975    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6" podStartSLOduration=1.827961854 podStartE2EDuration="1.827961854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:45.825501528 +0000 UTC m=+1456.031083580" watchObservedRunningTime="2026-11-03 10:53:45.827961854 +0000 UTC m=+1456.033543905"
Nov 03 10:53:46.151115 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:46.151034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:46.322219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:46.322189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:46.587491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:46.587448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:46.739596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:46.737459    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:46.824558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:46.824524    5817 generic.go:356] "Generic (PLEG): container finished" podID="d8c160ce-323b-477a-8dbf-dbab771c50c9" containerID="6e6b3d8c938fbfe79968a77fdb591095f130844f4c953de0c9ed649c44e7318c" exitCode=0
Nov 03 10:53:46.825035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:46.824797    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6" event={"ID":"d8c160ce-323b-477a-8dbf-dbab771c50c9","Type":"ContainerDied","Data":"6e6b3d8c938fbfe79968a77fdb591095f130844f4c953de0c9ed649c44e7318c"}
Nov 03 10:53:47.456055 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:47.456014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:47.982736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:47.982715    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:48.115510 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.115438    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c160ce-323b-477a-8dbf-dbab771c50c9-kube-api-access-w2dw8\" (UniqueName: \"kubernetes.io/projected/d8c160ce-323b-477a-8dbf-dbab771c50c9-kube-api-access-w2dw8\") pod \"d8c160ce-323b-477a-8dbf-dbab771c50c9\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") "
Nov 03 10:53:48.115703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.115522    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-openshift-ca\") pod \"d8c160ce-323b-477a-8dbf-dbab771c50c9\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") "
Nov 03 10:53:48.115780 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.115766    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-xks-ca\") pod \"d8c160ce-323b-477a-8dbf-dbab771c50c9\" (UID: \"d8c160ce-323b-477a-8dbf-dbab771c50c9\") "
Nov 03 10:53:48.115987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.115956    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-openshift-ca" pod "d8c160ce-323b-477a-8dbf-dbab771c50c9" (UID: "d8c160ce-323b-477a-8dbf-dbab771c50c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:48.116361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.116342    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:48.117987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.117957    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-xks-ca" pod "d8c160ce-323b-477a-8dbf-dbab771c50c9" (UID: "d8c160ce-323b-477a-8dbf-dbab771c50c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:48.118229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.118190    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c160ce-323b-477a-8dbf-dbab771c50c9-kube-api-access-w2dw8" pod "d8c160ce-323b-477a-8dbf-dbab771c50c9" (UID: "d8c160ce-323b-477a-8dbf-dbab771c50c9"). InnerVolumeSpecName "kube-api-access-w2dw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:48.218123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.218094    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c160ce-323b-477a-8dbf-dbab771c50c9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:48.218123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.218121    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2dw8\" (UniqueName: \"kubernetes.io/projected/d8c160ce-323b-477a-8dbf-dbab771c50c9-kube-api-access-w2dw8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:48.251366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.251327    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:48.841765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.841723    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6" event={"ID":"d8c160ce-323b-477a-8dbf-dbab771c50c9","Type":"ContainerDied","Data":"d75626945be377e69891de6505d894da74e5938771cc53a9d1d94ca015fc2738"}
Nov 03 10:53:48.841972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.841788    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75626945be377e69891de6505d894da74e5938771cc53a9d1d94ca015fc2738"
Nov 03 10:53:48.841972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.841790    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"
Nov 03 10:53:48.958151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:48.958110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:49.229883 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:49.229847    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:49.414642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:49.414608    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:49.862348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:49.862300    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:50.778068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:50.778025    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:50.936610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:50.933176    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:51.287567 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.287534    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"]
Nov 03 10:53:51.288355 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.288337    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d8c160ce-323b-477a-8dbf-dbab771c50c9" containerName="revoke-keys"
Nov 03 10:53:51.288402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.288358    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c160ce-323b-477a-8dbf-dbab771c50c9" containerName="revoke-keys"
Nov 03 10:53:51.288522 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.288512    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c160ce-323b-477a-8dbf-dbab771c50c9" containerName="revoke-keys"
Nov 03 10:53:51.292817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.292798    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.296179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.296155    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"]
Nov 03 10:53:51.297744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.297722    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:53:51.297835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.297731    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:53:51.472886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.472855    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.473149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.472930    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.473256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.473154    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgzq\" (UniqueName: \"kubernetes.io/projected/7c0ef350-2202-4d4b-9af1-13c0e1a40136-kube-api-access-khgzq\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.574416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.574333    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgzq\" (UniqueName: \"kubernetes.io/projected/7c0ef350-2202-4d4b-9af1-13c0e1a40136-kube-api-access-khgzq\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.574416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.574385    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.574789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.574480    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.575040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.575011    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.577169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.577142    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.581769 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.581744    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgzq\" (UniqueName: \"kubernetes.io/projected/7c0ef350-2202-4d4b-9af1-13c0e1a40136-kube-api-access-khgzq\") pod \"maas-api-revoke-sub-19eea206-j5fpg\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.608973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.608950    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:51.677209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.677173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:51.904759 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:53:51.904696    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0ef350_2202_4d4b_9af1_13c0e1a40136.slice/crio-6989f85669b856ee218945bf25c59500a64758cc70e69707e811a31bd359cabb WatchSource:0}: Error finding container 6989f85669b856ee218945bf25c59500a64758cc70e69707e811a31bd359cabb: Status 404 returned error can't find the container with id 6989f85669b856ee218945bf25c59500a64758cc70e69707e811a31bd359cabb
Nov 03 10:53:51.905331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:51.905041    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"]
Nov 03 10:53:52.038506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.038470    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:52.064733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.064704    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"]
Nov 03 10:53:52.069957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.069941    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.077983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.076972    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-35afa6-dockercfg-9jblj\""
Nov 03 10:53:52.077983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.077014    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-35afa6\""
Nov 03 10:53:52.081808 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.081784    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"]
Nov 03 10:53:52.190177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.190138    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905ebd8-52c3-411f-9a10-1a13c401e2ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b\" (UID: \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.190653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.190626    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvsq\" (UniqueName: \"kubernetes.io/projected/4905ebd8-52c3-411f-9a10-1a13c401e2ec-kube-api-access-gxvsq\") pod \"payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b\" (UID: \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.221603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.217775    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:52.297595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.291914    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905ebd8-52c3-411f-9a10-1a13c401e2ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b\" (UID: \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.297595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.292154    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvsq\" (UniqueName: \"kubernetes.io/projected/4905ebd8-52c3-411f-9a10-1a13c401e2ec-kube-api-access-gxvsq\") pod \"payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b\" (UID: \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.297595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.293337    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905ebd8-52c3-411f-9a10-1a13c401e2ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b\" (UID: \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.317616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.315348    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvsq\" (UniqueName: \"kubernetes.io/projected/4905ebd8-52c3-411f-9a10-1a13c401e2ec-kube-api-access-gxvsq\") pod \"payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b\" (UID: \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.384729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.384697    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.424617 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:53:52.423940    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:53:52.649129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.649096    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:52.681739 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.681712    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"]
Nov 03 10:53:52.682255 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:53:52.682228    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4905ebd8_52c3_411f_9a10_1a13c401e2ec.slice/crio-063c40be03a6d1cafe014da4d98c9abd0205ea8f185ee9562273c6547d2c5b99 WatchSource:0}: Error finding container 063c40be03a6d1cafe014da4d98c9abd0205ea8f185ee9562273c6547d2c5b99: Status 404 returned error can't find the container with id 063c40be03a6d1cafe014da4d98c9abd0205ea8f185ee9562273c6547d2c5b99
Nov 03 10:53:52.873316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.873284    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg" event={"ID":"7c0ef350-2202-4d4b-9af1-13c0e1a40136","Type":"ContainerStarted","Data":"d6d3d82b5d3bd3ed70ff47225a9ab54f170303b631b987f073b382347fdfc366"}
Nov 03 10:53:52.873495 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.873336    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg" event={"ID":"7c0ef350-2202-4d4b-9af1-13c0e1a40136","Type":"ContainerStarted","Data":"6989f85669b856ee218945bf25c59500a64758cc70e69707e811a31bd359cabb"}
Nov 03 10:53:52.875198 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.875171    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b" event={"ID":"4905ebd8-52c3-411f-9a10-1a13c401e2ec","Type":"ContainerStarted","Data":"2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f"}
Nov 03 10:53:52.875336 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.875219    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b" event={"ID":"4905ebd8-52c3-411f-9a10-1a13c401e2ec","Type":"ContainerStarted","Data":"063c40be03a6d1cafe014da4d98c9abd0205ea8f185ee9562273c6547d2c5b99"}
Nov 03 10:53:52.875336 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.875245    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:53:52.913154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.913106    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg" podStartSLOduration=1.913087717 podStartE2EDuration="1.913087717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:52.907982357 +0000 UTC m=+1463.113564409" watchObservedRunningTime="2026-11-03 10:53:52.913087717 +0000 UTC m=+1463.118669767"
Nov 03 10:53:52.946881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:52.946827    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b" podStartSLOduration=0.946809994 podStartE2EDuration="946.809994ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:52.942179624 +0000 UTC m=+1463.147761680" watchObservedRunningTime="2026-11-03 10:53:52.946809994 +0000 UTC m=+1463.152392047"
Nov 03 10:53:53.447939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:53.447911    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:53.889125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:53.888972    5817 generic.go:356] "Generic (PLEG): container finished" podID="7c0ef350-2202-4d4b-9af1-13c0e1a40136" containerID="d6d3d82b5d3bd3ed70ff47225a9ab54f170303b631b987f073b382347fdfc366" exitCode=0
Nov 03 10:53:53.889352 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:53.889292    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg" event={"ID":"7c0ef350-2202-4d4b-9af1-13c0e1a40136","Type":"ContainerDied","Data":"d6d3d82b5d3bd3ed70ff47225a9ab54f170303b631b987f073b382347fdfc366"}
Nov 03 10:53:53.972591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:53.972545    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:54.119595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:54.115959    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:54.733265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:54.728742    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:55.062541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.062519    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:55.155042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.155010    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-openshift-ca\") pod \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") "
Nov 03 10:53:55.155216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.155114    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-xks-ca\") pod \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") "
Nov 03 10:53:55.155216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.155200    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0ef350-2202-4d4b-9af1-13c0e1a40136-kube-api-access-khgzq\" (UniqueName: \"kubernetes.io/projected/7c0ef350-2202-4d4b-9af1-13c0e1a40136-kube-api-access-khgzq\") pod \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\" (UID: \"7c0ef350-2202-4d4b-9af1-13c0e1a40136\") "
Nov 03 10:53:55.155338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.155289    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-openshift-ca" pod "7c0ef350-2202-4d4b-9af1-13c0e1a40136" (UID: "7c0ef350-2202-4d4b-9af1-13c0e1a40136"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:55.155678 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.155660    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:55.157382 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.157362    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-xks-ca" pod "7c0ef350-2202-4d4b-9af1-13c0e1a40136" (UID: "7c0ef350-2202-4d4b-9af1-13c0e1a40136"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:55.157496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.157477    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0ef350-2202-4d4b-9af1-13c0e1a40136-kube-api-access-khgzq" pod "7c0ef350-2202-4d4b-9af1-13c0e1a40136" (UID: "7c0ef350-2202-4d4b-9af1-13c0e1a40136"). InnerVolumeSpecName "kube-api-access-khgzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:55.257149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.257120    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0ef350-2202-4d4b-9af1-13c0e1a40136-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:55.257149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.257146    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgzq\" (UniqueName: \"kubernetes.io/projected/7c0ef350-2202-4d4b-9af1-13c0e1a40136-kube-api-access-khgzq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:55.489996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.489963    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:55.929162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.929133    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"
Nov 03 10:53:55.929305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.929150    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg" event={"ID":"7c0ef350-2202-4d4b-9af1-13c0e1a40136","Type":"ContainerDied","Data":"6989f85669b856ee218945bf25c59500a64758cc70e69707e811a31bd359cabb"}
Nov 03 10:53:55.929305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:55.929209    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6989f85669b856ee218945bf25c59500a64758cc70e69707e811a31bd359cabb"
Nov 03 10:53:56.172515 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:56.172463    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:56.895100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:56.895064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:57.588862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:57.588820    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:57.744518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:57.744485    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:58.674258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:58.674204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:53:59.361492 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:53:59.361457    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:00.274709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:00.274673    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:01.093676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.093646    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"]
Nov 03 10:54:01.095398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.095372    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7c0ef350-2202-4d4b-9af1-13c0e1a40136" containerName="revoke-keys"
Nov 03 10:54:01.095589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.095556    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0ef350-2202-4d4b-9af1-13c0e1a40136" containerName="revoke-keys"
Nov 03 10:54:01.095891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.095877    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0ef350-2202-4d4b-9af1-13c0e1a40136" containerName="revoke-keys"
Nov 03 10:54:01.102988 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.100776    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"]
Nov 03 10:54:01.103194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.100905    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.166228 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.166197    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"]
Nov 03 10:54:01.166594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.166558    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"]
Nov 03 10:54:01.166745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.166503    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.249511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.249444    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.249848 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.249827    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.249992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.249971    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.250118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.250101    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.250267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.250251    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.250424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.250408    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.250546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.250531    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.250695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.250676    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.250849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.250832    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vsg\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-kube-api-access-w6vsg\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.250990 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.250976    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.251133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.251118    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.251253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.251237    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.251415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.251400    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.251586 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.251550    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.251784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.251767    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.251921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.251903    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.252046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.252029    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38216033-6b9e-4efe-8ae7-5e6b947349e9-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.252205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.252190    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrq2\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-kube-api-access-ltrq2\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.353655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353381    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.353655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353440    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.353655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353503    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.353655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353558    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.353655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353623    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.353655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353655    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353686    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38216033-6b9e-4efe-8ae7-5e6b947349e9-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353747    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrq2\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-kube-api-access-ltrq2\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353781    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353848    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353879    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353910    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353950    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.353996    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.354025    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.354059    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.354121    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vsg\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-kube-api-access-w6vsg\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.354459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.354184    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.371545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.359321    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.371545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.361010    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.371545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.361668    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.371545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.361909    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.371545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.362507    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38216033-6b9e-4efe-8ae7-5e6b947349e9-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.372400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.372369    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.372644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.372616    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.372758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.372654    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.372982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.372957    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.376602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.374617    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.376602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.375066    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.376602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.375475    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.376602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.375909    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.376885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.376799    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.378182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.378158    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.379038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.379005    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.380062 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.380040    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrq2\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-kube-api-access-ltrq2\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:01.380880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.380860    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vsg\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-kube-api-access-w6vsg\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.469028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.468998    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:01.496812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:01.496791    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:02.350640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.344347    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:02.646411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.631121    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:54:02.658914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.658860    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:02.659131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.659109    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:02.659262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.659246    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:02.660194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.659279    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"]
Nov 03 10:54:02.784260 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.783600    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"]
Nov 03 10:54:02.784951 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:02.784928    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38216033_6b9e_4efe_8ae7_5e6b947349e9.slice/crio-9b6d5af73f75e733678ad80c3499482a084a02b4c8b6dbf6998eb787c9f35079 WatchSource:0}: Error finding container 9b6d5af73f75e733678ad80c3499482a084a02b4c8b6dbf6998eb787c9f35079: Status 404 returned error can't find the container with id 9b6d5af73f75e733678ad80c3499482a084a02b4c8b6dbf6998eb787c9f35079
Nov 03 10:54:02.789719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.788787    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:02.789719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.788872    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:02.789719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:02.788917    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:03.666741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:03.666704    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" event={"ID":"c1d5fdd1-e359-49dc-93a2-01a188e52f24","Type":"ContainerStarted","Data":"357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806"}
Nov 03 10:54:03.667194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:03.666764    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" event={"ID":"c1d5fdd1-e359-49dc-93a2-01a188e52f24","Type":"ContainerStarted","Data":"67ef6c41e9b3f3e5273251989261b0300e4fe1526f1647491d786ad5df232418"}
Nov 03 10:54:03.668528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:03.668503    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" event={"ID":"38216033-6b9e-4efe-8ae7-5e6b947349e9","Type":"ContainerStarted","Data":"6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb"}
Nov 03 10:54:03.668661 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:03.668547    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" event={"ID":"38216033-6b9e-4efe-8ae7-5e6b947349e9","Type":"ContainerStarted","Data":"9b6d5af73f75e733678ad80c3499482a084a02b4c8b6dbf6998eb787c9f35079"}
Nov 03 10:54:03.706784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:03.706712    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podStartSLOduration=2.706693737 podStartE2EDuration="2.706693737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:03.684711565 +0000 UTC m=+1473.890293617" watchObservedRunningTime="2026-11-03 10:54:03.706693737 +0000 UTC m=+1473.912275788"
Nov 03 10:54:03.726677 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:03.723722    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podStartSLOduration=2.723704259 podStartE2EDuration="2.723704259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:03.723074383 +0000 UTC m=+1473.928656435" watchObservedRunningTime="2026-11-03 10:54:03.723704259 +0000 UTC m=+1473.929286309"
Nov 03 10:54:03.966945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:03.966914    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:54:04.156182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:04.156054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:04.471299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:04.471264    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:04.471485 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:04.471366    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:54:04.471485 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:04.471403    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 10:54:04.501596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:04.500488    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:04.501596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:04.500633    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 10:54:04.501596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:04.500691    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 10:54:05.490595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:05.484982    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:54:05.490595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:05.485055    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 10:54:05.502595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:05.499921    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 10:54:05.502595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:05.499976    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 10:54:05.963967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:05.963934    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:06.470633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:06.470595    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:54:06.470813 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:06.470661    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 10:54:06.499994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:06.499949    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 10:54:06.500550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:06.500518    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 10:54:06.679915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:06.679880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:07.470548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:07.470516    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:54:07.470835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:07.470595    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 10:54:07.508563 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:07.508530    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 10:54:07.509198 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:07.509151    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 10:54:07.564686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:07.564652    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:08.361924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:08.361883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:08.475598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:08.474972    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:54:08.475598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:08.475033    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 10:54:08.507599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:08.506974    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 10:54:08.507599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:08.507034    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 10:54:09.184316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:09.184268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:09.470486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:09.470386    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:54:09.470486 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:09.470455    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 10:54:09.498133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:09.498093    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 10:54:09.498341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:09.498176    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 10:54:10.116108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:10.116051    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:10.469822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:10.469789    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:54:10.470307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:10.469848    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 10:54:10.813768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:10.813677    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:10.946268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:10.946234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:11.109262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.109176    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:11.110591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.110545    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:11.112186 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.112162    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 10:54:11.301493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.301457    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:11.446078 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.446046    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:11.493599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.491882    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:11.493599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.493318    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:11.496847 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.496824    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 10:54:11.970243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:11.970178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:12.113769 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:12.113729    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:12.625364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:12.625330    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:12.769189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:12.769157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:13.263369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:13.263335    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:14.008205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.008176    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"]
Nov 03 10:54:14.012300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.012278    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.017607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.015565    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:54:14.017607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.016936    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:54:14.020807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.020775    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"]
Nov 03 10:54:14.059236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.059204    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcbtk\" (UniqueName: \"kubernetes.io/projected/d0c96ddb-fb91-47f8-aa22-4ab952c86199-kube-api-access-fcbtk\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.059490 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.059466    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.059612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.059511    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.088623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.084881    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:14.174858 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.174818    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.175064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.174897    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.175064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.174966    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcbtk\" (UniqueName: \"kubernetes.io/projected/d0c96ddb-fb91-47f8-aa22-4ab952c86199-kube-api-access-fcbtk\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.176699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.176673    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.178854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.178830    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.184323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.184298    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcbtk\" (UniqueName: \"kubernetes.io/projected/d0c96ddb-fb91-47f8-aa22-4ab952c86199-kube-api-access-fcbtk\") pod \"maas-api-revoke-sub-a6a5e72d-bk5s7\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.331459 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.331367    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:14.651439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.651415    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"]
Nov 03 10:54:14.652061 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:14.652035    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c96ddb_fb91_47f8_aa22_4ab952c86199.slice/crio-c3bad4b9ac4a0e887a54c23e2c889b86782f32b1e82d9ce3981ef1950fc936cb WatchSource:0}: Error finding container c3bad4b9ac4a0e887a54c23e2c889b86782f32b1e82d9ce3981ef1950fc936cb: Status 404 returned error can't find the container with id c3bad4b9ac4a0e887a54c23e2c889b86782f32b1e82d9ce3981ef1950fc936cb
Nov 03 10:54:14.870618 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:14.870587    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7" event={"ID":"d0c96ddb-fb91-47f8-aa22-4ab952c86199","Type":"ContainerStarted","Data":"c3bad4b9ac4a0e887a54c23e2c889b86782f32b1e82d9ce3981ef1950fc936cb"}
Nov 03 10:54:15.224195 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:15.219810    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:15.630953 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:15.630920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:15.823933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:15.823890    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:15.881847 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:15.881762    5817 generic.go:356] "Generic (PLEG): container finished" podID="d0c96ddb-fb91-47f8-aa22-4ab952c86199" containerID="48643a71a18f248387593cbf2d66869004bc8a5bc072aaad63acc1f670f1fc16" exitCode=0
Nov 03 10:54:15.882036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:15.881850    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7" event={"ID":"d0c96ddb-fb91-47f8-aa22-4ab952c86199","Type":"ContainerDied","Data":"48643a71a18f248387593cbf2d66869004bc8a5bc072aaad63acc1f670f1fc16"}
Nov 03 10:54:16.072458 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.072422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:16.781867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.781833    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:16.808781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.808750    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"]
Nov 03 10:54:16.815491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.814814    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:16.820122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.820100    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"]
Nov 03 10:54:16.820759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.820738    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8bbb6e\""
Nov 03 10:54:16.916683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.916402    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a31ed33-e02b-4e82-a4ac-413575c79312-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg\" (UID: \"6a31ed33-e02b-4e82-a4ac-413575c79312\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:16.916683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.916681    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzpq\" (UniqueName: \"kubernetes.io/projected/6a31ed33-e02b-4e82-a4ac-413575c79312-kube-api-access-hhzpq\") pod \"payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg\" (UID: \"6a31ed33-e02b-4e82-a4ac-413575c79312\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:16.939084 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:16.939050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:17.019607 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.019493    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a31ed33-e02b-4e82-a4ac-413575c79312-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg\" (UID: \"6a31ed33-e02b-4e82-a4ac-413575c79312\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:17.019906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.019697    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzpq\" (UniqueName: \"kubernetes.io/projected/6a31ed33-e02b-4e82-a4ac-413575c79312-kube-api-access-hhzpq\") pod \"payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg\" (UID: \"6a31ed33-e02b-4e82-a4ac-413575c79312\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:17.022682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.021026    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a31ed33-e02b-4e82-a4ac-413575c79312-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg\" (UID: \"6a31ed33-e02b-4e82-a4ac-413575c79312\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:17.111120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.105754    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzpq\" (UniqueName: \"kubernetes.io/projected/6a31ed33-e02b-4e82-a4ac-413575c79312-kube-api-access-hhzpq\") pod \"payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg\" (UID: \"6a31ed33-e02b-4e82-a4ac-413575c79312\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:17.147207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.147149    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:17.219815 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.219260    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:17.323594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.323288    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c96ddb-fb91-47f8-aa22-4ab952c86199-kube-api-access-fcbtk\" (UniqueName: \"kubernetes.io/projected/d0c96ddb-fb91-47f8-aa22-4ab952c86199-kube-api-access-fcbtk\") pod \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") "
Nov 03 10:54:17.323594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.323448    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-openshift-ca\") pod \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") "
Nov 03 10:54:17.323594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.323543    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-xks-ca\") pod \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\" (UID: \"d0c96ddb-fb91-47f8-aa22-4ab952c86199\") "
Nov 03 10:54:17.335259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.325183    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-openshift-ca" pod "d0c96ddb-fb91-47f8-aa22-4ab952c86199" (UID: "d0c96ddb-fb91-47f8-aa22-4ab952c86199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:17.335259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.329115    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-xks-ca" pod "d0c96ddb-fb91-47f8-aa22-4ab952c86199" (UID: "d0c96ddb-fb91-47f8-aa22-4ab952c86199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:17.335259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.335199    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c96ddb-fb91-47f8-aa22-4ab952c86199-kube-api-access-fcbtk" pod "d0c96ddb-fb91-47f8-aa22-4ab952c86199" (UID: "d0c96ddb-fb91-47f8-aa22-4ab952c86199"). InnerVolumeSpecName "kube-api-access-fcbtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:17.425629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.425552    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcbtk\" (UniqueName: \"kubernetes.io/projected/d0c96ddb-fb91-47f8-aa22-4ab952c86199-kube-api-access-fcbtk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:17.425629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.425593    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:17.425629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.425607    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c96ddb-fb91-47f8-aa22-4ab952c86199-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:17.457660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.457633    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"]
Nov 03 10:54:17.458856 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:17.458832    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a31ed33_e02b_4e82_a4ac_413575c79312.slice/crio-0f6f635313cd678ccc42bcf2761df36aca67797edaf0e90eb38686a6e4a88031 WatchSource:0}: Error finding container 0f6f635313cd678ccc42bcf2761df36aca67797edaf0e90eb38686a6e4a88031: Status 404 returned error can't find the container with id 0f6f635313cd678ccc42bcf2761df36aca67797edaf0e90eb38686a6e4a88031
Nov 03 10:54:17.658465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.658376    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:17.916315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.916285    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"
Nov 03 10:54:17.917814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.917652    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7" event={"ID":"d0c96ddb-fb91-47f8-aa22-4ab952c86199","Type":"ContainerDied","Data":"c3bad4b9ac4a0e887a54c23e2c889b86782f32b1e82d9ce3981ef1950fc936cb"}
Nov 03 10:54:17.917814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.917711    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3bad4b9ac4a0e887a54c23e2c889b86782f32b1e82d9ce3981ef1950fc936cb"
Nov 03 10:54:17.919382 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.919352    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg" event={"ID":"6a31ed33-e02b-4e82-a4ac-413575c79312","Type":"ContainerStarted","Data":"267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3"}
Nov 03 10:54:17.919475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.919407    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg" event={"ID":"6a31ed33-e02b-4e82-a4ac-413575c79312","Type":"ContainerStarted","Data":"0f6f635313cd678ccc42bcf2761df36aca67797edaf0e90eb38686a6e4a88031"}
Nov 03 10:54:17.920380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.920363    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:17.938637 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:17.938595    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg" podStartSLOduration=1.938565651 podStartE2EDuration="1.938565651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:17.937695558 +0000 UTC m=+1488.143277611" watchObservedRunningTime="2026-11-03 10:54:17.938565651 +0000 UTC m=+1488.144147701"
Nov 03 10:54:18.265944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.265911    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"]
Nov 03 10:54:18.266535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.266522    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d0c96ddb-fb91-47f8-aa22-4ab952c86199" containerName="revoke-keys"
Nov 03 10:54:18.266625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.266538    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c96ddb-fb91-47f8-aa22-4ab952c86199" containerName="revoke-keys"
Nov 03 10:54:18.266692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.266681    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c96ddb-fb91-47f8-aa22-4ab952c86199" containerName="revoke-keys"
Nov 03 10:54:18.270439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.270413    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.285488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.282744    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:54:18.285488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.282790    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:54:18.324343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.324309    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"]
Nov 03 10:54:18.337741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.337710    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.337912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.337786    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.337972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.337900    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bhn\" (UniqueName: \"kubernetes.io/projected/87486b01-289a-4295-83a0-4b00ff2e44d8-kube-api-access-n4bhn\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.439696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.439668    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bhn\" (UniqueName: \"kubernetes.io/projected/87486b01-289a-4295-83a0-4b00ff2e44d8-kube-api-access-n4bhn\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.439862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.439847    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.439959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.439886    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.440464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.440444    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.442299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.442277    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.447381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.447361    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bhn\" (UniqueName: \"kubernetes.io/projected/87486b01-289a-4295-83a0-4b00ff2e44d8-kube-api-access-n4bhn\") pod \"maas-api-revoke-sub-989db9cc-jmd8d\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.558436 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.557874    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:18.611200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.611166    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:18.901910 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.901881    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"]
Nov 03 10:54:18.902090 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:18.902063    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87486b01_289a_4295_83a0_4b00ff2e44d8.slice/crio-b74b73554bb7ffcd9f435dd613e2748fd2d7e674be8901cbf0b805e31eb748c7 WatchSource:0}: Error finding container b74b73554bb7ffcd9f435dd613e2748fd2d7e674be8901cbf0b805e31eb748c7: Status 404 returned error can't find the container with id b74b73554bb7ffcd9f435dd613e2748fd2d7e674be8901cbf0b805e31eb748c7
Nov 03 10:54:18.931346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:18.931321    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d" event={"ID":"87486b01-289a-4295-83a0-4b00ff2e44d8","Type":"ContainerStarted","Data":"b74b73554bb7ffcd9f435dd613e2748fd2d7e674be8901cbf0b805e31eb748c7"}
Nov 03 10:54:19.254629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:19.254026    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:19.941178 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:19.941147    5817 generic.go:356] "Generic (PLEG): container finished" podID="87486b01-289a-4295-83a0-4b00ff2e44d8" containerID="700482ae49d420f7534273f6b56d69bf38fd406edd505d2537c8abe683ace4c4" exitCode=0
Nov 03 10:54:19.941624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:19.941206    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d" event={"ID":"87486b01-289a-4295-83a0-4b00ff2e44d8","Type":"ContainerDied","Data":"700482ae49d420f7534273f6b56d69bf38fd406edd505d2537c8abe683ace4c4"}
Nov 03 10:54:20.000781 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:20.000564    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:20.660963 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:20.660914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:21.120625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.120601    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:21.179796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.179768    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-openshift-ca\") pod \"87486b01-289a-4295-83a0-4b00ff2e44d8\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") "
Nov 03 10:54:21.179957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.179828    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87486b01-289a-4295-83a0-4b00ff2e44d8-kube-api-access-n4bhn\" (UniqueName: \"kubernetes.io/projected/87486b01-289a-4295-83a0-4b00ff2e44d8-kube-api-access-n4bhn\") pod \"87486b01-289a-4295-83a0-4b00ff2e44d8\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") "
Nov 03 10:54:21.180090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.180076    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-xks-ca\") pod \"87486b01-289a-4295-83a0-4b00ff2e44d8\" (UID: \"87486b01-289a-4295-83a0-4b00ff2e44d8\") "
Nov 03 10:54:21.180468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.180185    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-openshift-ca" pod "87486b01-289a-4295-83a0-4b00ff2e44d8" (UID: "87486b01-289a-4295-83a0-4b00ff2e44d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:21.180654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.180594    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.182224 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.182187    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87486b01-289a-4295-83a0-4b00ff2e44d8-kube-api-access-n4bhn" pod "87486b01-289a-4295-83a0-4b00ff2e44d8" (UID: "87486b01-289a-4295-83a0-4b00ff2e44d8"). InnerVolumeSpecName "kube-api-access-n4bhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:21.182733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.182707    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-xks-ca" pod "87486b01-289a-4295-83a0-4b00ff2e44d8" (UID: "87486b01-289a-4295-83a0-4b00ff2e44d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:21.281976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.281896    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bhn\" (UniqueName: \"kubernetes.io/projected/87486b01-289a-4295-83a0-4b00ff2e44d8-kube-api-access-n4bhn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.281976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.281931    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87486b01-289a-4295-83a0-4b00ff2e44d8-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.348057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.348014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:21.982665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.982622    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:21.986403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.984837    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"
Nov 03 10:54:21.986403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.984858    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d" event={"ID":"87486b01-289a-4295-83a0-4b00ff2e44d8","Type":"ContainerDied","Data":"b74b73554bb7ffcd9f435dd613e2748fd2d7e674be8901cbf0b805e31eb748c7"}
Nov 03 10:54:21.986403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:21.984913    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b74b73554bb7ffcd9f435dd613e2748fd2d7e674be8901cbf0b805e31eb748c7"
Nov 03 10:54:22.413668 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:54:22.413633    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:54:22.683554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:22.683524    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:22.820341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:22.820306    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:23.024275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:23.024193    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:54:23.652595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:23.651857    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:24.956594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:24.955609    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:25.730713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:25.730666    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:25.778598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:25.768678    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:25.787598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:25.782885    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.734595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:26.720559    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:26.938610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:26.927492    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:27.330183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:27.330146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:27.530219 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:27.530180    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:28.077381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.077348    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:28.287849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.287811    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:28.416183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.416065    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"]
Nov 03 10:54:28.418125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.418096    5817 cpu_manager.go:414] "containerMap: removing container" podUID="87486b01-289a-4295-83a0-4b00ff2e44d8" containerName="revoke-keys"
Nov 03 10:54:28.418289 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.418274    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87486b01-289a-4295-83a0-4b00ff2e44d8" containerName="revoke-keys"
Nov 03 10:54:28.418692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.418675    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="87486b01-289a-4295-83a0-4b00ff2e44d8" containerName="revoke-keys"
Nov 03 10:54:28.425951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.425926    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"]
Nov 03 10:54:28.426076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.426063    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.455883 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.455845    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.456125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.456086    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.456265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.456168    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.456265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.456207    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.456399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.456314    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1738b7-8733-48da-b0d7-5f3be1475436-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.456399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.456358    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.456550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.456431    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xgwm\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-kube-api-access-8xgwm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.456550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.456457    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.456550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.456494    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558003    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558079    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xgwm\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-kube-api-access-8xgwm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558178    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558216    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558386    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558412    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558561 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558458    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558561 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558481    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558561 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558540    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558684    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1738b7-8733-48da-b0d7-5f3be1475436-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.558814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.558763    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.559236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.559207    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.559520 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.559486    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.559681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.559589    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1738b7-8733-48da-b0d7-5f3be1475436-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.561238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.561213    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.561867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.561848    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.567708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.567684    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xgwm\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-kube-api-access-8xgwm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.568868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.568845    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:28.652183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.650554    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:28.773165 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:28.773138    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:29.073900 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.073875    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"]
Nov 03 10:54:29.074692 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:29.074658    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1738b7_8733_48da_b0d7_5f3be1475436.slice/crio-fb5907738945f5e9c2a967d1a3eb38b15b8f0d6b33cc9554ff2a19c87f913594 WatchSource:0}: Error finding container fb5907738945f5e9c2a967d1a3eb38b15b8f0d6b33cc9554ff2a19c87f913594: Status 404 returned error can't find the container with id fb5907738945f5e9c2a967d1a3eb38b15b8f0d6b33cc9554ff2a19c87f913594
Nov 03 10:54:29.077172 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.077142    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:29.077243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.077204    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:29.077243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.077232    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:29.259717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.259685    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" event={"ID":"bf1738b7-8733-48da-b0d7-5f3be1475436","Type":"ContainerStarted","Data":"8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e"}
Nov 03 10:54:29.260129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.259746    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" event={"ID":"bf1738b7-8733-48da-b0d7-5f3be1475436","Type":"ContainerStarted","Data":"fb5907738945f5e9c2a967d1a3eb38b15b8f0d6b33cc9554ff2a19c87f913594"}
Nov 03 10:54:29.281228 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.281181    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" podStartSLOduration=1.2811675390000001 podStartE2EDuration="1.281167539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:29.277663869 +0000 UTC m=+1499.483245920" watchObservedRunningTime="2026-11-03 10:54:29.281167539 +0000 UTC m=+1499.486749590"
Nov 03 10:54:29.675997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.675951    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:29.778600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.774633    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:29.944390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:29.944368    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:30.775733 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:30.775699    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 10:54:30.776142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:30.775754    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 10:54:30.821598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:30.821022    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:31.112643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.112047    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"]
Nov 03 10:54:31.127275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.126809    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"]
Nov 03 10:54:31.127275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.126933    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.130775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.130689    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:54:31.131845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.131816    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:54:31.241221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.241185    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.241437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.241347    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58pf\" (UniqueName: \"kubernetes.io/projected/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-kube-api-access-f58pf\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.241437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.241399    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.342925 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.342889    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.343123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.343016    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58pf\" (UniqueName: \"kubernetes.io/projected/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-kube-api-access-f58pf\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.343123 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.343059    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.343515 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.343491    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.345528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.345511    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.351021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.351001    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58pf\" (UniqueName: \"kubernetes.io/projected/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-kube-api-access-f58pf\") pod \"maas-api-revoke-sub-f31a69b8-v6fkn\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.371184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.371122    5817 scope.go:122] "RemoveContainer" containerID="ae31e1d6f825bde5b032f611c81f5e075fd00d8cc452296fbc51223a8be6fab5"
Nov 03 10:54:31.381200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.381176    5817 scope.go:122] "RemoveContainer" containerID="7ca4f3a878b96fd10c4bdea1c0c86c466f1f040d72b5f342c064849047d458ca"
Nov 03 10:54:31.398802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.398784    5817 scope.go:122] "RemoveContainer" containerID="03e0161c9305dfec25e3caeb2c17ac4cbe9ab33298fd91eb15d85d315fdf9656"
Nov 03 10:54:31.450483 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.450462    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:31.709592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.689062    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:31.831111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.806328    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 10:54:31.831111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.806386    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 10:54:31.831111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:31.827386    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"]
Nov 03 10:54:31.833714 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:31.833679    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e299c0_33d2_4e04_80a8_cd3588f5cbf2.slice/crio-c5945d97b0404f301a56341fa9749c3dd29b5d9ea06b16ae663cfd026a7cfb17 WatchSource:0}: Error finding container c5945d97b0404f301a56341fa9749c3dd29b5d9ea06b16ae663cfd026a7cfb17: Status 404 returned error can't find the container with id c5945d97b0404f301a56341fa9749c3dd29b5d9ea06b16ae663cfd026a7cfb17
Nov 03 10:54:32.297652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:32.293320    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn" event={"ID":"19e299c0-33d2-4e04-80a8-cd3588f5cbf2","Type":"ContainerStarted","Data":"c5945d97b0404f301a56341fa9749c3dd29b5d9ea06b16ae663cfd026a7cfb17"}
Nov 03 10:54:32.612599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:32.611843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:32.774329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:32.774292    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 10:54:32.774536 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:32.774346    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 10:54:33.323845 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:33.323813    5817 generic.go:356] "Generic (PLEG): container finished" podID="19e299c0-33d2-4e04-80a8-cd3588f5cbf2" containerID="d57cf0f6f190f37be4c280d6f2cff65874bdaf065620b35bcc0ea0a1602d3ae9" exitCode=0
Nov 03 10:54:33.324278 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:33.323895    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn" event={"ID":"19e299c0-33d2-4e04-80a8-cd3588f5cbf2","Type":"ContainerDied","Data":"d57cf0f6f190f37be4c280d6f2cff65874bdaf065620b35bcc0ea0a1602d3ae9"}
Nov 03 10:54:33.363205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:33.363171    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:33.774372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:33.774315    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 10:54:33.774546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:33.774393    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 10:54:34.066074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.065983    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:34.557180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.557158    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:34.639745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.639655    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-xks-ca\") pod \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") "
Nov 03 10:54:34.639745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.639744    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-kube-api-access-f58pf\" (UniqueName: \"kubernetes.io/projected/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-kube-api-access-f58pf\") pod \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") "
Nov 03 10:54:34.639996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.639896    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-openshift-ca\") pod \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\" (UID: \"19e299c0-33d2-4e04-80a8-cd3588f5cbf2\") "
Nov 03 10:54:34.640489 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.640467    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-openshift-ca" pod "19e299c0-33d2-4e04-80a8-cd3588f5cbf2" (UID: "19e299c0-33d2-4e04-80a8-cd3588f5cbf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:34.642848 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.642821    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-kube-api-access-f58pf" pod "19e299c0-33d2-4e04-80a8-cd3588f5cbf2" (UID: "19e299c0-33d2-4e04-80a8-cd3588f5cbf2"). InnerVolumeSpecName "kube-api-access-f58pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:34.643805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.643776    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-xks-ca" pod "19e299c0-33d2-4e04-80a8-cd3588f5cbf2" (UID: "19e299c0-33d2-4e04-80a8-cd3588f5cbf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:34.741233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.741200    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:34.741233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.741229    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58pf\" (UniqueName: \"kubernetes.io/projected/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-kube-api-access-f58pf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:34.741233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.741241    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e299c0-33d2-4e04-80a8-cd3588f5cbf2-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:34.773923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.773881    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 10:54:34.774089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.773943    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 10:54:34.964367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.964332    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"]
Nov 03 10:54:34.964973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.964948    5817 cpu_manager.go:414] "containerMap: removing container" podUID="19e299c0-33d2-4e04-80a8-cd3588f5cbf2" containerName="revoke-keys"
Nov 03 10:54:34.964973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.964971    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e299c0-33d2-4e04-80a8-cd3588f5cbf2" containerName="revoke-keys"
Nov 03 10:54:34.965184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.965135    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e299c0-33d2-4e04-80a8-cd3588f5cbf2" containerName="revoke-keys"
Nov 03 10:54:34.968824 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.968803    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:34.979606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:34.979556    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"]
Nov 03 10:54:35.047368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.045516    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.047368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.045654    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x9nn\" (UniqueName: \"kubernetes.io/projected/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-kube-api-access-5x9nn\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.047368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.045789    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.077742 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.077675    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:35.146798 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.146768    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.147005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.146870    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x9nn\" (UniqueName: \"kubernetes.io/projected/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-kube-api-access-5x9nn\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.147005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.146996    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.148061 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.148036    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.150162 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.150138    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.155056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.155033    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x9nn\" (UniqueName: \"kubernetes.io/projected/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-kube-api-access-5x9nn\") pod \"maas-api-revoke-sub-23f773ba-8tz7g\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.335204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.335120    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:35.342937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.342916    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"
Nov 03 10:54:35.343074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.342953    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn" event={"ID":"19e299c0-33d2-4e04-80a8-cd3588f5cbf2","Type":"ContainerDied","Data":"c5945d97b0404f301a56341fa9749c3dd29b5d9ea06b16ae663cfd026a7cfb17"}
Nov 03 10:54:35.343074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.342999    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5945d97b0404f301a56341fa9749c3dd29b5d9ea06b16ae663cfd026a7cfb17"
Nov 03 10:54:35.727905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.727881    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"]
Nov 03 10:54:35.797142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.787951    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 10:54:35.797142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:35.788027    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 10:54:36.018969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:36.018924    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:36.365168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:36.365064    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g" event={"ID":"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f","Type":"ContainerStarted","Data":"d1e110f1119596ee6853b943b23eab48106e737cc358788a049bc054d436627d"}
Nov 03 10:54:36.365330 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:36.365169    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g" event={"ID":"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f","Type":"ContainerStarted","Data":"58798b8b3513621a65c7f6dfb95009e57037b63a83487bc36df381707f45c8b4"}
Nov 03 10:54:36.386216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:36.386160    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g" podStartSLOduration=2.3861481270000002 podStartE2EDuration="2.386148127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:36.382474789 +0000 UTC m=+1506.588056840" watchObservedRunningTime="2026-11-03 10:54:36.386148127 +0000 UTC m=+1506.591730222"
Nov 03 10:54:36.778494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:36.778467    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:36.779149 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:36.779126    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:36.779615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:36.779599    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 10:54:36.840358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:36.840328    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:37.374935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:37.374902    5817 generic.go:356] "Generic (PLEG): container finished" podID="6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f" containerID="d1e110f1119596ee6853b943b23eab48106e737cc358788a049bc054d436627d" exitCode=0
Nov 03 10:54:37.375143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:37.374983    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g" event={"ID":"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f","Type":"ContainerDied","Data":"d1e110f1119596ee6853b943b23eab48106e737cc358788a049bc054d436627d"}
Nov 03 10:54:37.665206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:37.665118    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:37.805551 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:37.805510    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:38.037316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.037287    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:38.551367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.551341    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:38.624551 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.624505    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-xks-ca\") pod \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") "
Nov 03 10:54:38.624760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.624738    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-kube-api-access-5x9nn\" (UniqueName: \"kubernetes.io/projected/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-kube-api-access-5x9nn\") pod \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") "
Nov 03 10:54:38.624954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.624934    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-openshift-ca\") pod \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\" (UID: \"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f\") "
Nov 03 10:54:38.625379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.625338    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-openshift-ca" pod "6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f" (UID: "6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:38.625512 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.625495    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:38.627307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.627285    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-xks-ca" pod "6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f" (UID: "6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:38.627637 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.627611    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-kube-api-access-5x9nn" pod "6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f" (UID: "6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f"). InnerVolumeSpecName "kube-api-access-5x9nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:38.659424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.659390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:38.726477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.726447    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x9nn\" (UniqueName: \"kubernetes.io/projected/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-kube-api-access-5x9nn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:38.726477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:38.726480    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:39.397112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:39.397079    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"
Nov 03 10:54:39.397605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:39.397132    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g" event={"ID":"6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f","Type":"ContainerDied","Data":"58798b8b3513621a65c7f6dfb95009e57037b63a83487bc36df381707f45c8b4"}
Nov 03 10:54:39.397605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:39.397203    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58798b8b3513621a65c7f6dfb95009e57037b63a83487bc36df381707f45c8b4"
Nov 03 10:54:39.456616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:39.452122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:40.089598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:40.087429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:40.658937 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:40.658902    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"]
Nov 03 10:54:40.779665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:40.779621    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:40.914526 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:40.914440    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:41.080454 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:41.080293    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:41.782096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:41.782060    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:42.576018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:42.575945    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:42.777899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:42.777769    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:43.463174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:43.463140    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:44.250856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.250825    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"]
Nov 03 10:54:44.251419 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.251405    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f" containerName="revoke-keys"
Nov 03 10:54:44.251484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.251420    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f" containerName="revoke-keys"
Nov 03 10:54:44.251535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.251526    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f" containerName="revoke-keys"
Nov 03 10:54:44.255364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.255340    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.262194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.262169    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"]
Nov 03 10:54:44.263103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.263086    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:54:44.264342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.264324    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:54:44.365720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.365688    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:44.429555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.429506    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpjx\" (UniqueName: \"kubernetes.io/projected/a102aa71-6f66-4838-8172-ba70d5262a28-kube-api-access-kcpjx\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.429818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.429635    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.429818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.429666    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.531467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.531390    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpjx\" (UniqueName: \"kubernetes.io/projected/a102aa71-6f66-4838-8172-ba70d5262a28-kube-api-access-kcpjx\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.531467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.531449    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.531957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.531483    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.532376 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.532347    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.534527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.534502    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.539042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.539020    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpjx\" (UniqueName: \"kubernetes.io/projected/a102aa71-6f66-4838-8172-ba70d5262a28-kube-api-access-kcpjx\") pod \"maas-api-revoke-sub-faeb35e0-8j5mk\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.577379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.577358    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:44.867519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:44.867483    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"]
Nov 03 10:54:44.867678 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:44.867549    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda102aa71_6f66_4838_8172_ba70d5262a28.slice/crio-be301e313581dcd743304abc5d365a1b5bda0d7f3c9a1417cda88de7d23773f6 WatchSource:0}: Error finding container be301e313581dcd743304abc5d365a1b5bda0d7f3c9a1417cda88de7d23773f6: Status 404 returned error can't find the container with id be301e313581dcd743304abc5d365a1b5bda0d7f3c9a1417cda88de7d23773f6
Nov 03 10:54:45.217366 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:45.217340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:45.456357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:45.456274    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk" event={"ID":"a102aa71-6f66-4838-8172-ba70d5262a28","Type":"ContainerStarted","Data":"b966a7a4530adae64834dfc49665412988ccab84ab1e50fa1aa62e7e842ac5f3"}
Nov 03 10:54:45.456357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:45.456325    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk" event={"ID":"a102aa71-6f66-4838-8172-ba70d5262a28","Type":"ContainerStarted","Data":"be301e313581dcd743304abc5d365a1b5bda0d7f3c9a1417cda88de7d23773f6"}
Nov 03 10:54:45.477676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:45.476953    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk" podStartSLOduration=1.4769357859999999 podStartE2EDuration="1.476935786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:45.47024993 +0000 UTC m=+1515.675831980" watchObservedRunningTime="2026-11-03 10:54:45.476935786 +0000 UTC m=+1515.682517838"
Nov 03 10:54:45.659902 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:45.659855    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg" podUID="6a31ed33-e02b-4e82-a4ac-413575c79312" containerName="payload-processing" containerID="cri-o://267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3" gracePeriod=25
Nov 03 10:54:45.673372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:45.673349    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:45.815865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:45.815726    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:46.179992 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.179963    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:46.191248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.191221    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a31ed33-e02b-4e82-a4ac-413575c79312-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a31ed33-e02b-4e82-a4ac-413575c79312-plugins-config-volume\") pod \"6a31ed33-e02b-4e82-a4ac-413575c79312\" (UID: \"6a31ed33-e02b-4e82-a4ac-413575c79312\") "
Nov 03 10:54:46.191392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.191325    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a31ed33-e02b-4e82-a4ac-413575c79312-kube-api-access-hhzpq\" (UniqueName: \"kubernetes.io/projected/6a31ed33-e02b-4e82-a4ac-413575c79312-kube-api-access-hhzpq\") pod \"6a31ed33-e02b-4e82-a4ac-413575c79312\" (UID: \"6a31ed33-e02b-4e82-a4ac-413575c79312\") "
Nov 03 10:54:46.191605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.191566    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a31ed33-e02b-4e82-a4ac-413575c79312-plugins-config-volume" pod "6a31ed33-e02b-4e82-a4ac-413575c79312" (UID: "6a31ed33-e02b-4e82-a4ac-413575c79312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:46.191946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.191930    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a31ed33-e02b-4e82-a4ac-413575c79312-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:46.193545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.193521    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a31ed33-e02b-4e82-a4ac-413575c79312-kube-api-access-hhzpq" pod "6a31ed33-e02b-4e82-a4ac-413575c79312" (UID: "6a31ed33-e02b-4e82-a4ac-413575c79312"). InnerVolumeSpecName "kube-api-access-hhzpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:46.293367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.293329    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzpq\" (UniqueName: \"kubernetes.io/projected/6a31ed33-e02b-4e82-a4ac-413575c79312-kube-api-access-hhzpq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:46.443919 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.443880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:46.469105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.469073    5817 generic.go:356] "Generic (PLEG): container finished" podID="6a31ed33-e02b-4e82-a4ac-413575c79312" containerID="267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3" exitCode=0
Nov 03 10:54:46.469423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.469111    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg" event={"ID":"6a31ed33-e02b-4e82-a4ac-413575c79312","Type":"ContainerDied","Data":"267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3"}
Nov 03 10:54:46.469423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.469181    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg" event={"ID":"6a31ed33-e02b-4e82-a4ac-413575c79312","Type":"ContainerDied","Data":"0f6f635313cd678ccc42bcf2761df36aca67797edaf0e90eb38686a6e4a88031"}
Nov 03 10:54:46.469423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.469178    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"
Nov 03 10:54:46.469423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.469201    5817 scope.go:122] "RemoveContainer" containerID="267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3"
Nov 03 10:54:46.471954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.471932    5817 generic.go:356] "Generic (PLEG): container finished" podID="a102aa71-6f66-4838-8172-ba70d5262a28" containerID="b966a7a4530adae64834dfc49665412988ccab84ab1e50fa1aa62e7e842ac5f3" exitCode=0
Nov 03 10:54:46.472129 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.472004    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk" event={"ID":"a102aa71-6f66-4838-8172-ba70d5262a28","Type":"ContainerDied","Data":"b966a7a4530adae64834dfc49665412988ccab84ab1e50fa1aa62e7e842ac5f3"}
Nov 03 10:54:46.483141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.483122    5817 scope.go:122] "RemoveContainer" containerID="267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3"
Nov 03 10:54:46.483440 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:54:46.483421    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3\": container with ID starting with 267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3 not found: ID does not exist" containerID="267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3"
Nov 03 10:54:46.483491 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.483449    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3"} err="failed to get container status \"267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3\": rpc error: code = NotFound desc = could not find container \"267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3\": container with ID starting with 267ab11d5e3a08857baa0ff080bcf22478aa008d84a126ae90d63938d8249ff3 not found: ID does not exist"
Nov 03 10:54:46.518927 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.518846    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"]
Nov 03 10:54:46.528689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:46.525876    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-798b84b989-4bhzg"]
Nov 03 10:54:47.158111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.152235    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:47.812949 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.812917    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:47.875419 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.875392    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a102aa71-6f66-4838-8172-ba70d5262a28-kube-api-access-kcpjx\" (UniqueName: \"kubernetes.io/projected/a102aa71-6f66-4838-8172-ba70d5262a28-kube-api-access-kcpjx\") pod \"a102aa71-6f66-4838-8172-ba70d5262a28\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") "
Nov 03 10:54:47.875623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.875435    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-openshift-ca\") pod \"a102aa71-6f66-4838-8172-ba70d5262a28\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") "
Nov 03 10:54:47.875623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.875505    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-xks-ca\") pod \"a102aa71-6f66-4838-8172-ba70d5262a28\" (UID: \"a102aa71-6f66-4838-8172-ba70d5262a28\") "
Nov 03 10:54:47.875853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.875821    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-openshift-ca" pod "a102aa71-6f66-4838-8172-ba70d5262a28" (UID: "a102aa71-6f66-4838-8172-ba70d5262a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:47.876434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.876413    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:47.877818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.877797    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-xks-ca" pod "a102aa71-6f66-4838-8172-ba70d5262a28" (UID: "a102aa71-6f66-4838-8172-ba70d5262a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:47.877818 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.877809    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a102aa71-6f66-4838-8172-ba70d5262a28-kube-api-access-kcpjx" pod "a102aa71-6f66-4838-8172-ba70d5262a28" (UID: "a102aa71-6f66-4838-8172-ba70d5262a28"). InnerVolumeSpecName "kube-api-access-kcpjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:47.904971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.904944    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a31ed33-e02b-4e82-a4ac-413575c79312" path="/var/lib/kubelet/pods/6a31ed33-e02b-4e82-a4ac-413575c79312/volumes"
Nov 03 10:54:47.978163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.978138    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpjx\" (UniqueName: \"kubernetes.io/projected/a102aa71-6f66-4838-8172-ba70d5262a28-kube-api-access-kcpjx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:47.978163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:47.978163    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a102aa71-6f66-4838-8172-ba70d5262a28-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:48.080245 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:48.080159    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:48.500594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:48.500447    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"
Nov 03 10:54:48.501182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:48.500889    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk" event={"ID":"a102aa71-6f66-4838-8172-ba70d5262a28","Type":"ContainerDied","Data":"be301e313581dcd743304abc5d365a1b5bda0d7f3c9a1417cda88de7d23773f6"}
Nov 03 10:54:48.501182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:48.500971    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be301e313581dcd743304abc5d365a1b5bda0d7f3c9a1417cda88de7d23773f6"
Nov 03 10:54:50.141605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.141437    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"]
Nov 03 10:54:50.142133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.141713    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg" podUID="4c363a74-0ed9-47ba-9902-0345252cda60" containerName="discovery" containerID="cri-o://f3da7c43780d558de06b890a4d4a0775da9c4f58aef1f7a1dc48eb07c4f5cd46" gracePeriod=30
Nov 03 10:54:50.581543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.579660    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:54:50.581543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.580914    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.616892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.616851    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.668598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.644267    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:50.693925 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.693248    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.739597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.737381    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:54:50.739597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.737685    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:54:50.739597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.737866    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:54:50.739597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.738955    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.799735 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.769773    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.802885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:50.802399    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:51.655948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.626681    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:51.655948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.627408    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:54:51.655948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.628454    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:54:51.655948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.628511    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:54:51.744699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.742955    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:51.744699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.743221    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:54:51.744699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.743746    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:54:51.744699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.743805    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:54:51.770598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.770461    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:54:51.986601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:51.982126    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:54:52.076245 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.076212    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:52.085494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.085465    5817 generic.go:356] "Generic (PLEG): container finished" podID="4c363a74-0ed9-47ba-9902-0345252cda60" containerID="f3da7c43780d558de06b890a4d4a0775da9c4f58aef1f7a1dc48eb07c4f5cd46" exitCode=0
Nov 03 10:54:52.099296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.085636    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg" event={"ID":"4c363a74-0ed9-47ba-9902-0345252cda60","Type":"ContainerDied","Data":"f3da7c43780d558de06b890a4d4a0775da9c4f58aef1f7a1dc48eb07c4f5cd46"}
Nov 03 10:54:52.099296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.096402    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:54:52.099296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.096464    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:54:52.116042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.115213    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:54:52.257925 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.257893    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-dns-cert\") pod \"4c363a74-0ed9-47ba-9902-0345252cda60\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") "
Nov 03 10:54:52.258085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.257961    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-ca-configmap\") pod \"4c363a74-0ed9-47ba-9902-0345252cda60\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") "
Nov 03 10:54:52.258085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.258022    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-cacerts\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-cacerts\") pod \"4c363a74-0ed9-47ba-9902-0345252cda60\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") "
Nov 03 10:54:52.258209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.258101    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-kube-api-access-v9khs\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-kube-api-access-v9khs\") pod \"4c363a74-0ed9-47ba-9902-0345252cda60\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") "
Nov 03 10:54:52.258546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.258296    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c363a74-0ed9-47ba-9902-0345252cda60-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c363a74-0ed9-47ba-9902-0345252cda60-local-certs\") pod \"4c363a74-0ed9-47ba-9902-0345252cda60\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") "
Nov 03 10:54:52.258546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.258340    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-istio-token\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-istio-token\") pod \"4c363a74-0ed9-47ba-9902-0345252cda60\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") "
Nov 03 10:54:52.258546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.258383    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-kubeconfig\") pod \"4c363a74-0ed9-47ba-9902-0345252cda60\" (UID: \"4c363a74-0ed9-47ba-9902-0345252cda60\") "
Nov 03 10:54:52.258546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.258390    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-ca-configmap" pod "4c363a74-0ed9-47ba-9902-0345252cda60" (UID: "4c363a74-0ed9-47ba-9902-0345252cda60"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:52.259037 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.259009    5817 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-ca-configmap\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:52.261311 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.261243    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-istio-token" pod "4c363a74-0ed9-47ba-9902-0345252cda60" (UID: "4c363a74-0ed9-47ba-9902-0345252cda60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:52.261801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.261775    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-dns-cert" pod "4c363a74-0ed9-47ba-9902-0345252cda60" (UID: "4c363a74-0ed9-47ba-9902-0345252cda60"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:52.261944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.261864    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-cacerts" pod "4c363a74-0ed9-47ba-9902-0345252cda60" (UID: "4c363a74-0ed9-47ba-9902-0345252cda60"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:52.263361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.263326    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-kube-api-access-v9khs" pod "4c363a74-0ed9-47ba-9902-0345252cda60" (UID: "4c363a74-0ed9-47ba-9902-0345252cda60"). InnerVolumeSpecName "kube-api-access-v9khs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:52.263475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.263445    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-kubeconfig" pod "4c363a74-0ed9-47ba-9902-0345252cda60" (UID: "4c363a74-0ed9-47ba-9902-0345252cda60"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:52.263546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.263509    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c363a74-0ed9-47ba-9902-0345252cda60-local-certs" pod "4c363a74-0ed9-47ba-9902-0345252cda60" (UID: "4c363a74-0ed9-47ba-9902-0345252cda60"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:54:52.362716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.362618    5817 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-cacerts\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:52.362716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.362664    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9khs\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-kube-api-access-v9khs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:52.362716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.362685    5817 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c363a74-0ed9-47ba-9902-0345252cda60-local-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:52.362716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.362702    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c363a74-0ed9-47ba-9902-0345252cda60-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:52.362716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.362716    5817 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-kubeconfig\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:52.363104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:52.362730    5817 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c363a74-0ed9-47ba-9902-0345252cda60-istio-csr-dns-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:52.424031 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:54:52.423989    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:54:53.111168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.111138    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"
Nov 03 10:54:53.111692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.111157    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg" event={"ID":"4c363a74-0ed9-47ba-9902-0345252cda60","Type":"ContainerDied","Data":"17efca0260bc6c53f2ebb1c914983400b1eb54ca520660839595b23d00274256"}
Nov 03 10:54:53.111867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.111846    5817 scope.go:122] "RemoveContainer" containerID="f3da7c43780d558de06b890a4d4a0775da9c4f58aef1f7a1dc48eb07c4f5cd46"
Nov 03 10:54:53.140969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.140946    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"]
Nov 03 10:54:53.148356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.148334    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-94kvg"]
Nov 03 10:54:53.437403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.437367    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"]
Nov 03 10:54:53.438221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438195    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6a31ed33-e02b-4e82-a4ac-413575c79312" containerName="payload-processing"
Nov 03 10:54:53.438221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438217    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a31ed33-e02b-4e82-a4ac-413575c79312" containerName="payload-processing"
Nov 03 10:54:53.438395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438247    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a102aa71-6f66-4838-8172-ba70d5262a28" containerName="revoke-keys"
Nov 03 10:54:53.438395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438254    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a102aa71-6f66-4838-8172-ba70d5262a28" containerName="revoke-keys"
Nov 03 10:54:53.438395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438278    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4c363a74-0ed9-47ba-9902-0345252cda60" containerName="discovery"
Nov 03 10:54:53.438395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438285    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c363a74-0ed9-47ba-9902-0345252cda60" containerName="discovery"
Nov 03 10:54:53.438534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438413    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a31ed33-e02b-4e82-a4ac-413575c79312" containerName="payload-processing"
Nov 03 10:54:53.438534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438427    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a102aa71-6f66-4838-8172-ba70d5262a28" containerName="revoke-keys"
Nov 03 10:54:53.438534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.438438    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c363a74-0ed9-47ba-9902-0345252cda60" containerName="discovery"
Nov 03 10:54:53.443675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.443641    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.447616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.447355    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:54:53.447616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.447473    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:54:53.449052 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.449029    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"]
Nov 03 10:54:53.579898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.579863    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbtst\" (UniqueName: \"kubernetes.io/projected/12101dab-8bc8-4346-915a-3e271c775db2-kube-api-access-fbtst\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.580097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.579926    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.580171 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.580152    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.681780 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.681748    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.682229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.681882    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.682327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.682234    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtst\" (UniqueName: \"kubernetes.io/projected/12101dab-8bc8-4346-915a-3e271c775db2-kube-api-access-fbtst\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.682558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.682536    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.690986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.690929    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.692517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.692490    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtst\" (UniqueName: \"kubernetes.io/projected/12101dab-8bc8-4346-915a-3e271c775db2-kube-api-access-fbtst\") pod \"maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.763975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.763945    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:53.935615 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:53.935081    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c363a74-0ed9-47ba-9902-0345252cda60" path="/var/lib/kubelet/pods/4c363a74-0ed9-47ba-9902-0345252cda60/volumes"
Nov 03 10:54:54.068805 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:54.068772    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"]
Nov 03 10:54:54.068977 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:54.068958    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12101dab_8bc8_4346_915a_3e271c775db2.slice/crio-299dc9622f39ba61820b8840e7c067b3fe4fc27b3038ff466e1ab8336c8dd99e WatchSource:0}: Error finding container 299dc9622f39ba61820b8840e7c067b3fe4fc27b3038ff466e1ab8336c8dd99e: Status 404 returned error can't find the container with id 299dc9622f39ba61820b8840e7c067b3fe4fc27b3038ff466e1ab8336c8dd99e
Nov 03 10:54:54.120803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:54.120772    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz" event={"ID":"12101dab-8bc8-4346-915a-3e271c775db2","Type":"ContainerStarted","Data":"299dc9622f39ba61820b8840e7c067b3fe4fc27b3038ff466e1ab8336c8dd99e"}
Nov 03 10:54:54.592955 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:54.592920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:55.132939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:55.132907    5817 generic.go:356] "Generic (PLEG): container finished" podID="12101dab-8bc8-4346-915a-3e271c775db2" containerID="72e3885803b04e8dcdeafb30092f69949d18cce72ded2e6383ad4a799ff4affa" exitCode=0
Nov 03 10:54:55.133354 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:55.132993    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz" event={"ID":"12101dab-8bc8-4346-915a-3e271c775db2","Type":"ContainerDied","Data":"72e3885803b04e8dcdeafb30092f69949d18cce72ded2e6383ad4a799ff4affa"}
Nov 03 10:54:55.686043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:55.686007    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:56.317644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.317616    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:56.420906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.420825    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-openshift-ca\") pod \"12101dab-8bc8-4346-915a-3e271c775db2\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") "
Nov 03 10:54:56.420906 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.420878    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-xks-ca\") pod \"12101dab-8bc8-4346-915a-3e271c775db2\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") "
Nov 03 10:54:56.421132 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.420915    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12101dab-8bc8-4346-915a-3e271c775db2-kube-api-access-fbtst\" (UniqueName: \"kubernetes.io/projected/12101dab-8bc8-4346-915a-3e271c775db2-kube-api-access-fbtst\") pod \"12101dab-8bc8-4346-915a-3e271c775db2\" (UID: \"12101dab-8bc8-4346-915a-3e271c775db2\") "
Nov 03 10:54:56.421251 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.421224    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-openshift-ca" pod "12101dab-8bc8-4346-915a-3e271c775db2" (UID: "12101dab-8bc8-4346-915a-3e271c775db2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:56.421403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.421382    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:56.423046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.423020    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-xks-ca" pod "12101dab-8bc8-4346-915a-3e271c775db2" (UID: "12101dab-8bc8-4346-915a-3e271c775db2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:56.423180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.423163    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12101dab-8bc8-4346-915a-3e271c775db2-kube-api-access-fbtst" pod "12101dab-8bc8-4346-915a-3e271c775db2" (UID: "12101dab-8bc8-4346-915a-3e271c775db2"). InnerVolumeSpecName "kube-api-access-fbtst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:56.522587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.522544    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12101dab-8bc8-4346-915a-3e271c775db2-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:56.522587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.522584    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbtst\" (UniqueName: \"kubernetes.io/projected/12101dab-8bc8-4346-915a-3e271c775db2-kube-api-access-fbtst\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:56.744030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:56.743319    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:57.023369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.023282    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"]
Nov 03 10:54:57.029448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.029423    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9781a50d-6skj6"]
Nov 03 10:54:57.150029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.150004    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"
Nov 03 10:54:57.150193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.150039    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz" event={"ID":"12101dab-8bc8-4346-915a-3e271c775db2","Type":"ContainerDied","Data":"299dc9622f39ba61820b8840e7c067b3fe4fc27b3038ff466e1ab8336c8dd99e"}
Nov 03 10:54:57.150193 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.150082    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="299dc9622f39ba61820b8840e7c067b3fe4fc27b3038ff466e1ab8336c8dd99e"
Nov 03 10:54:57.360378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.360304    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"]
Nov 03 10:54:57.360970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.360952    5817 cpu_manager.go:414] "containerMap: removing container" podUID="12101dab-8bc8-4346-915a-3e271c775db2" containerName="revoke-keys"
Nov 03 10:54:57.360970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.360970    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12101dab-8bc8-4346-915a-3e271c775db2" containerName="revoke-keys"
Nov 03 10:54:57.361076 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.361067    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="12101dab-8bc8-4346-915a-3e271c775db2" containerName="revoke-keys"
Nov 03 10:54:57.364759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.364741    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.369752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.369723    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:54:57.369882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.369847    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:54:57.372718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.372697    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"]
Nov 03 10:54:57.435772 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.435740    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.435948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.435925    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.436066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.436045    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw65j\" (UniqueName: \"kubernetes.io/projected/257a5ec2-87c8-4eda-b090-161ab102a5ab-kube-api-access-rw65j\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.537001    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.537204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.537079    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.537204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.537116    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw65j\" (UniqueName: \"kubernetes.io/projected/257a5ec2-87c8-4eda-b090-161ab102a5ab-kube-api-access-rw65j\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.537693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.537667    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.539642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.539626    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.544871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.544847    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw65j\" (UniqueName: \"kubernetes.io/projected/257a5ec2-87c8-4eda-b090-161ab102a5ab-kube-api-access-rw65j\") pod \"maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.682169 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.682144    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:54:57.905426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.905385    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68457f8-5de8-4e6b-adfb-e4597ea247a7" path="/var/lib/kubelet/pods/f68457f8-5de8-4e6b-adfb-e4597ea247a7/volumes"
Nov 03 10:54:57.979237 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:54:57.979204    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257a5ec2_87c8_4eda_b090_161ab102a5ab.slice/crio-67ae7fb7d780b058569ad28ccd933f1c47481ab83a238cf5b4cb06c548c87412 WatchSource:0}: Error finding container 67ae7fb7d780b058569ad28ccd933f1c47481ab83a238cf5b4cb06c548c87412: Status 404 returned error can't find the container with id 67ae7fb7d780b058569ad28ccd933f1c47481ab83a238cf5b4cb06c548c87412
Nov 03 10:54:57.979561 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:57.979540    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"]
Nov 03 10:54:58.126080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:58.126033    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:58.168302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:58.167813    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6" event={"ID":"257a5ec2-87c8-4eda-b090-161ab102a5ab","Type":"ContainerStarted","Data":"67ae7fb7d780b058569ad28ccd933f1c47481ab83a238cf5b4cb06c548c87412"}
Nov 03 10:54:59.138614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:59.138559    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:54:59.183814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:59.183780    5817 generic.go:356] "Generic (PLEG): container finished" podID="257a5ec2-87c8-4eda-b090-161ab102a5ab" containerID="de524aed76e26badeb5a30ef181783bc38e31d3793b8e8f79c8ae291e26b695f" exitCode=0
Nov 03 10:54:59.183979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:59.183866    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6" event={"ID":"257a5ec2-87c8-4eda-b090-161ab102a5ab","Type":"ContainerDied","Data":"de524aed76e26badeb5a30ef181783bc38e31d3793b8e8f79c8ae291e26b695f"}
Nov 03 10:54:59.763090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:54:59.763055    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:00.360875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.360848    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:55:00.381718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.381691    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-openshift-ca\") pod \"257a5ec2-87c8-4eda-b090-161ab102a5ab\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") "
Nov 03 10:55:00.381884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.381819    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257a5ec2-87c8-4eda-b090-161ab102a5ab-kube-api-access-rw65j\" (UniqueName: \"kubernetes.io/projected/257a5ec2-87c8-4eda-b090-161ab102a5ab-kube-api-access-rw65j\") pod \"257a5ec2-87c8-4eda-b090-161ab102a5ab\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") "
Nov 03 10:55:00.381954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.381916    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-xks-ca\") pod \"257a5ec2-87c8-4eda-b090-161ab102a5ab\" (UID: \"257a5ec2-87c8-4eda-b090-161ab102a5ab\") "
Nov 03 10:55:00.382112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.382082    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-openshift-ca" pod "257a5ec2-87c8-4eda-b090-161ab102a5ab" (UID: "257a5ec2-87c8-4eda-b090-161ab102a5ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:00.382737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.382710    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:00.384212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.384182    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257a5ec2-87c8-4eda-b090-161ab102a5ab-kube-api-access-rw65j" pod "257a5ec2-87c8-4eda-b090-161ab102a5ab" (UID: "257a5ec2-87c8-4eda-b090-161ab102a5ab"). InnerVolumeSpecName "kube-api-access-rw65j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:00.384296 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.384254    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-xks-ca" pod "257a5ec2-87c8-4eda-b090-161ab102a5ab" (UID: "257a5ec2-87c8-4eda-b090-161ab102a5ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:00.484694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.484618    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw65j\" (UniqueName: \"kubernetes.io/projected/257a5ec2-87c8-4eda-b090-161ab102a5ab-kube-api-access-rw65j\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:00.484694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.484653    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257a5ec2-87c8-4eda-b090-161ab102a5ab-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:00.658317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:00.658279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:01.202069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:01.202036    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"
Nov 03 10:55:01.202069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:01.202041    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6" event={"ID":"257a5ec2-87c8-4eda-b090-161ab102a5ab","Type":"ContainerDied","Data":"67ae7fb7d780b058569ad28ccd933f1c47481ab83a238cf5b4cb06c548c87412"}
Nov 03 10:55:01.202365 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:01.202086    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67ae7fb7d780b058569ad28ccd933f1c47481ab83a238cf5b4cb06c548c87412"
Nov 03 10:55:01.463662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:01.454490    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:02.230637 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:02.230594    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:02.367956 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:02.367914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:03.059682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:03.057696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:03.583744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:03.583711    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"]
Nov 03 10:55:03.693493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:03.693452    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:03.820103 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:03.820072    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:03.952983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:03.951342    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:04.866950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:04.866920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:05.032375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:05.032348    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2f7kn_99ada003-4afe-4019-8dab-6c9dadf4b888/payload-processing/0.log"
Nov 03 10:55:05.684084 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:05.684045    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:06.402141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:06.402095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:06.685508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:06.685009    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:06.820597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:06.820151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:07.047752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:07.047616    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"]
Nov 03 10:55:07.053090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:07.053063    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-35afa6-rjlfz"]
Nov 03 10:55:07.434317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:07.434282    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:07.568656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:07.568623    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:07.809382 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:07.809292    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:07.911544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:07.911461    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12101dab-8bc8-4346-915a-3e271c775db2" path="/var/lib/kubelet/pods/12101dab-8bc8-4346-915a-3e271c775db2/volumes"
Nov 03 10:55:08.585269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:08.585225    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b" podUID="4905ebd8-52c3-411f-9a10-1a13c401e2ec" containerName="payload-pre-processing" containerID="cri-o://2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f" gracePeriod=25
Nov 03 10:55:08.650640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:08.650514    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:08.894197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:08.894159    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:55:08.928679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:08.928650    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4905ebd8-52c3-411f-9a10-1a13c401e2ec-kube-api-access-gxvsq\" (UniqueName: \"kubernetes.io/projected/4905ebd8-52c3-411f-9a10-1a13c401e2ec-kube-api-access-gxvsq\") pod \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\" (UID: \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\") "
Nov 03 10:55:08.928987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:08.928968    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4905ebd8-52c3-411f-9a10-1a13c401e2ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905ebd8-52c3-411f-9a10-1a13c401e2ec-plugins-config-volume\") pod \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\" (UID: \"4905ebd8-52c3-411f-9a10-1a13c401e2ec\") "
Nov 03 10:55:08.929360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:08.929336    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4905ebd8-52c3-411f-9a10-1a13c401e2ec-plugins-config-volume" pod "4905ebd8-52c3-411f-9a10-1a13c401e2ec" (UID: "4905ebd8-52c3-411f-9a10-1a13c401e2ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:08.929549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:08.929533    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905ebd8-52c3-411f-9a10-1a13c401e2ec-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:08.931543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:08.931519    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4905ebd8-52c3-411f-9a10-1a13c401e2ec-kube-api-access-gxvsq" pod "4905ebd8-52c3-411f-9a10-1a13c401e2ec" (UID: "4905ebd8-52c3-411f-9a10-1a13c401e2ec"). InnerVolumeSpecName "kube-api-access-gxvsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:09.030638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.030602    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxvsq\" (UniqueName: \"kubernetes.io/projected/4905ebd8-52c3-411f-9a10-1a13c401e2ec-kube-api-access-gxvsq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:09.273398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.273349    5817 generic.go:356] "Generic (PLEG): container finished" podID="4905ebd8-52c3-411f-9a10-1a13c401e2ec" containerID="2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f" exitCode=0
Nov 03 10:55:09.273398 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.273390    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b" event={"ID":"4905ebd8-52c3-411f-9a10-1a13c401e2ec","Type":"ContainerDied","Data":"2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f"}
Nov 03 10:55:09.273716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.273418    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"
Nov 03 10:55:09.273716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.273447    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b" event={"ID":"4905ebd8-52c3-411f-9a10-1a13c401e2ec","Type":"ContainerDied","Data":"063c40be03a6d1cafe014da4d98c9abd0205ea8f185ee9562273c6547d2c5b99"}
Nov 03 10:55:09.273716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.273482    5817 scope.go:122] "RemoveContainer" containerID="2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f"
Nov 03 10:55:09.286070 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.286047    5817 scope.go:122] "RemoveContainer" containerID="2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f"
Nov 03 10:55:09.286345 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:55:09.286327    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f\": container with ID starting with 2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f not found: ID does not exist" containerID="2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f"
Nov 03 10:55:09.286403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.286356    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f"} err="failed to get container status \"2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f\": rpc error: code = NotFound desc = could not find container \"2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f\": container with ID starting with 2e2b43f430eedacb176664cc4888d3c14d0526d8e74a6e433621cf7bd1f02a3f not found: ID does not exist"
Nov 03 10:55:09.296879 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.296850    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"]
Nov 03 10:55:09.302917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.302891    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-35afa6-6b9f65cc9-6bk9b"]
Nov 03 10:55:09.544327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.544232    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:09.603439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.603406    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:55:09.904892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:09.904819    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4905ebd8-52c3-411f-9a10-1a13c401e2ec" path="/var/lib/kubelet/pods/4905ebd8-52c3-411f-9a10-1a13c401e2ec/volumes"
Nov 03 10:55:10.519300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:10.519263    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:11.689601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:11.682061    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:11.751598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:11.750641    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:11.751777 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:11.751647    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:11.772597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:11.763760    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:55:11.789595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:11.786349    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:11.986656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:11.984658    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.062593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.052640    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.080598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.063319    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.171706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.171668    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:55:12.171971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.171943    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.238282 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.237925    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.550796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.550688    5817 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:55:12.551106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.551078    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.752697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.752648    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.753588 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.753264    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:55:12.773879 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.754179    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:55:12.774111 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.774095    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:55:12.793825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.793762    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.860470 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.842649    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.905889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:12.905854    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:13.139321 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:13.139231    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:55:13.288598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:13.286638    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:55:13.484294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:13.484257    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:13.626807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:13.626769    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:13.779695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:13.779613    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:14.183177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.183139    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"]
Nov 03 10:55:14.183746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.183729    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4905ebd8-52c3-411f-9a10-1a13c401e2ec" containerName="payload-pre-processing"
Nov 03 10:55:14.183746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.183745    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4905ebd8-52c3-411f-9a10-1a13c401e2ec" containerName="payload-pre-processing"
Nov 03 10:55:14.183914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.183783    5817 cpu_manager.go:414] "containerMap: removing container" podUID="257a5ec2-87c8-4eda-b090-161ab102a5ab" containerName="revoke-keys"
Nov 03 10:55:14.183914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.183789    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257a5ec2-87c8-4eda-b090-161ab102a5ab" containerName="revoke-keys"
Nov 03 10:55:14.183914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.183868    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="257a5ec2-87c8-4eda-b090-161ab102a5ab" containerName="revoke-keys"
Nov 03 10:55:14.183914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.183881    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4905ebd8-52c3-411f-9a10-1a13c401e2ec" containerName="payload-pre-processing"
Nov 03 10:55:14.188715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.188678    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.195117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.195092    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"]
Nov 03 10:55:14.216853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.216830    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"]
Nov 03 10:55:14.221035 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.221016    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.228657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.228628    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"]
Nov 03 10:55:14.249530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.249509    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65np\" (UniqueName: \"kubernetes.io/projected/37cfba9a-3846-47ac-8f76-536d5e5d685d-kube-api-access-r65np\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.249672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.249544    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lld6t\" (UniqueName: \"kubernetes.io/projected/15465683-12e9-4bb3-9592-5e5abe524338-kube-api-access-lld6t\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.249720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.249691    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.249761 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.249751    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.249828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.249814    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.249894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.249883    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.256493 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.256474    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:55:14.256636 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.256548    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:55:14.350935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.350905    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.351098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.351000    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65np\" (UniqueName: \"kubernetes.io/projected/37cfba9a-3846-47ac-8f76-536d5e5d685d-kube-api-access-r65np\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.351098 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.351026    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lld6t\" (UniqueName: \"kubernetes.io/projected/15465683-12e9-4bb3-9592-5e5abe524338-kube-api-access-lld6t\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.351172 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.351108    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.351223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.351177    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.351223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.351200    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.351839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.351815    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.351946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.351876    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.353444 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.353425    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.353622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.353604    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.358562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.358543    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65np\" (UniqueName: \"kubernetes.io/projected/37cfba9a-3846-47ac-8f76-536d5e5d685d-kube-api-access-r65np\") pod \"maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.358795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.358774    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lld6t\" (UniqueName: \"kubernetes.io/projected/15465683-12e9-4bb3-9592-5e5abe524338-kube-api-access-lld6t\") pod \"maas-api-revoke-sub-59ad0d35-cvm29\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.518340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.518257    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:14.544597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.544549    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:14.724427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.724216    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:14.801542 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:55:14.801506    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15465683_12e9_4bb3_9592_5e5abe524338.slice/crio-b7cb4c312989b8936a1b5add852b92aaa9bb4a2be950594d94c0fbb762c3db6b WatchSource:0}: Error finding container b7cb4c312989b8936a1b5add852b92aaa9bb4a2be950594d94c0fbb762c3db6b: Status 404 returned error can't find the container with id b7cb4c312989b8936a1b5add852b92aaa9bb4a2be950594d94c0fbb762c3db6b
Nov 03 10:55:14.803847 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.803822    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"]
Nov 03 10:55:14.844915 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:55:14.844887    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cfba9a_3846_47ac_8f76_536d5e5d685d.slice/crio-81f79080d2e6d8729661b581d3313cb921451367abfdaac83a169e3e4f3a340a WatchSource:0}: Error finding container 81f79080d2e6d8729661b581d3313cb921451367abfdaac83a169e3e4f3a340a: Status 404 returned error can't find the container with id 81f79080d2e6d8729661b581d3313cb921451367abfdaac83a169e3e4f3a340a
Nov 03 10:55:14.845229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:14.845209    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"]
Nov 03 10:55:15.250236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:15.250205    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2f7kn_99ada003-4afe-4019-8dab-6c9dadf4b888/payload-processing/0.log"
Nov 03 10:55:15.398621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:15.398521    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29" event={"ID":"15465683-12e9-4bb3-9592-5e5abe524338","Type":"ContainerStarted","Data":"06be730dec2db25a68f03b564e170bfbfb6328282a2181b65ec19f75ecd4a684"}
Nov 03 10:55:15.398621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:15.398604    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29" event={"ID":"15465683-12e9-4bb3-9592-5e5abe524338","Type":"ContainerStarted","Data":"b7cb4c312989b8936a1b5add852b92aaa9bb4a2be950594d94c0fbb762c3db6b"}
Nov 03 10:55:15.400358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:15.400333    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds" event={"ID":"37cfba9a-3846-47ac-8f76-536d5e5d685d","Type":"ContainerStarted","Data":"b48d042299d17ec2b37a6c765a83bdf903cdcd15b91a3b7ca865a505a52be4ff"}
Nov 03 10:55:15.400472 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:15.400382    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds" event={"ID":"37cfba9a-3846-47ac-8f76-536d5e5d685d","Type":"ContainerStarted","Data":"81f79080d2e6d8729661b581d3313cb921451367abfdaac83a169e3e4f3a340a"}
Nov 03 10:55:15.415070 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:15.415030    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29" podStartSLOduration=1.415018058 podStartE2EDuration="1.415018058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:55:15.41293955 +0000 UTC m=+1545.618521601" watchObservedRunningTime="2026-11-03 10:55:15.415018058 +0000 UTC m=+1545.620600108"
Nov 03 10:55:15.428780 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:15.428734    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds" podStartSLOduration=1.428716182 podStartE2EDuration="1.428716182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:55:15.426922046 +0000 UTC m=+1545.632504096" watchObservedRunningTime="2026-11-03 10:55:15.428716182 +0000 UTC m=+1545.634298239"
Nov 03 10:55:15.700985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:15.700949    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:16.119157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.119074    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"]
Nov 03 10:55:16.123337 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.123311    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.143896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.143870    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"]
Nov 03 10:55:16.176016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.175978    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.176175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.176122    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l22qb\" (UniqueName: \"kubernetes.io/projected/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-kube-api-access-l22qb\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.176175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.176153    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.278215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.278178    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.278399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.278301    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22qb\" (UniqueName: \"kubernetes.io/projected/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-kube-api-access-l22qb\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.278399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.278333    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.279005 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.278982    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.280763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.280741    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.286514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.286494    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l22qb\" (UniqueName: \"kubernetes.io/projected/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-kube-api-access-l22qb\") pod \"maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.412155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.412064    5817 generic.go:356] "Generic (PLEG): container finished" podID="15465683-12e9-4bb3-9592-5e5abe524338" containerID="06be730dec2db25a68f03b564e170bfbfb6328282a2181b65ec19f75ecd4a684" exitCode=0
Nov 03 10:55:16.412320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.412147    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29" event={"ID":"15465683-12e9-4bb3-9592-5e5abe524338","Type":"ContainerDied","Data":"06be730dec2db25a68f03b564e170bfbfb6328282a2181b65ec19f75ecd4a684"}
Nov 03 10:55:16.414740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.414712    5817 generic.go:356] "Generic (PLEG): container finished" podID="37cfba9a-3846-47ac-8f76-536d5e5d685d" containerID="b48d042299d17ec2b37a6c765a83bdf903cdcd15b91a3b7ca865a505a52be4ff" exitCode=0
Nov 03 10:55:16.414865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.414750    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds" event={"ID":"37cfba9a-3846-47ac-8f76-536d5e5d685d","Type":"ContainerDied","Data":"b48d042299d17ec2b37a6c765a83bdf903cdcd15b91a3b7ca865a505a52be4ff"}
Nov 03 10:55:16.463585 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.463548    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:16.521074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.521034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:16.801854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:16.801812    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"]
Nov 03 10:55:17.174544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.174356    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:17.429182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.429123    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg" event={"ID":"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e","Type":"ContainerStarted","Data":"ffcd4080ab3296e52d6e920fe843b4180cfa5ca5f054e8c63a99af6e995b4e13"}
Nov 03 10:55:17.429346 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.429210    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg" event={"ID":"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e","Type":"ContainerStarted","Data":"566163dbac6eb7516fc778a27de382098dd78fb5200ba52dc8ad727fcc976b88"}
Nov 03 10:55:17.451922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.451865    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg" podStartSLOduration=1.451846899 podStartE2EDuration="1.451846899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:55:17.447507925 +0000 UTC m=+1547.653089975" watchObservedRunningTime="2026-11-03 10:55:17.451846899 +0000 UTC m=+1547.657428952"
Nov 03 10:55:17.641427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.641399    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:17.681132 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.681073    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:17.703069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.703037    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-openshift-ca\") pod \"37cfba9a-3846-47ac-8f76-536d5e5d685d\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") "
Nov 03 10:55:17.703463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.703110    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15465683-12e9-4bb3-9592-5e5abe524338-kube-api-access-lld6t\" (UniqueName: \"kubernetes.io/projected/15465683-12e9-4bb3-9592-5e5abe524338-kube-api-access-lld6t\") pod \"15465683-12e9-4bb3-9592-5e5abe524338\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") "
Nov 03 10:55:17.703463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.703240    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-xks-ca\") pod \"15465683-12e9-4bb3-9592-5e5abe524338\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") "
Nov 03 10:55:17.703463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.703268    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-openshift-ca\") pod \"15465683-12e9-4bb3-9592-5e5abe524338\" (UID: \"15465683-12e9-4bb3-9592-5e5abe524338\") "
Nov 03 10:55:17.703999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.703967    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-openshift-ca" pod "15465683-12e9-4bb3-9592-5e5abe524338" (UID: "15465683-12e9-4bb3-9592-5e5abe524338"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:17.704099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.703750    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-openshift-ca" pod "37cfba9a-3846-47ac-8f76-536d5e5d685d" (UID: "37cfba9a-3846-47ac-8f76-536d5e5d685d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:17.704488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.704463    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-xks-ca\") pod \"37cfba9a-3846-47ac-8f76-536d5e5d685d\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") "
Nov 03 10:55:17.704621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.704600    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cfba9a-3846-47ac-8f76-536d5e5d685d-kube-api-access-r65np\" (UniqueName: \"kubernetes.io/projected/37cfba9a-3846-47ac-8f76-536d5e5d685d-kube-api-access-r65np\") pod \"37cfba9a-3846-47ac-8f76-536d5e5d685d\" (UID: \"37cfba9a-3846-47ac-8f76-536d5e5d685d\") "
Nov 03 10:55:17.705238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.705211    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:17.705333 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.705243    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:17.707056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.707021    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15465683-12e9-4bb3-9592-5e5abe524338-kube-api-access-lld6t" pod "15465683-12e9-4bb3-9592-5e5abe524338" (UID: "15465683-12e9-4bb3-9592-5e5abe524338"). InnerVolumeSpecName "kube-api-access-lld6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:17.710603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.710544    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-xks-ca" pod "15465683-12e9-4bb3-9592-5e5abe524338" (UID: "15465683-12e9-4bb3-9592-5e5abe524338"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:17.710603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.710585    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cfba9a-3846-47ac-8f76-536d5e5d685d-kube-api-access-r65np" pod "37cfba9a-3846-47ac-8f76-536d5e5d685d" (UID: "37cfba9a-3846-47ac-8f76-536d5e5d685d"). InnerVolumeSpecName "kube-api-access-r65np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:17.710972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.710953    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-xks-ca" pod "37cfba9a-3846-47ac-8f76-536d5e5d685d" (UID: "37cfba9a-3846-47ac-8f76-536d5e5d685d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:17.806587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.806545    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15465683-12e9-4bb3-9592-5e5abe524338-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:17.806709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.806612    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfba9a-3846-47ac-8f76-536d5e5d685d-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:17.806709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.806624    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r65np\" (UniqueName: \"kubernetes.io/projected/37cfba9a-3846-47ac-8f76-536d5e5d685d-kube-api-access-r65np\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:17.806709 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.806635    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lld6t\" (UniqueName: \"kubernetes.io/projected/15465683-12e9-4bb3-9592-5e5abe524338-kube-api-access-lld6t\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:17.862741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:17.862705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:18.230886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.230802    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"]
Nov 03 10:55:18.231300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.230994    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy" containerID="cri-o://22335baf83eef623ce60c4332cf236a8bd188af309cd0de381dc0418b7600892" gracePeriod=30
Nov 03 10:55:18.439143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.439113    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"
Nov 03 10:55:18.439324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.439150    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29" event={"ID":"15465683-12e9-4bb3-9592-5e5abe524338","Type":"ContainerDied","Data":"b7cb4c312989b8936a1b5add852b92aaa9bb4a2be950594d94c0fbb762c3db6b"}
Nov 03 10:55:18.439324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.439198    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7cb4c312989b8936a1b5add852b92aaa9bb4a2be950594d94c0fbb762c3db6b"
Nov 03 10:55:18.440629 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.440606    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"
Nov 03 10:55:18.440765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.440606    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds" event={"ID":"37cfba9a-3846-47ac-8f76-536d5e5d685d","Type":"ContainerDied","Data":"81f79080d2e6d8729661b581d3313cb921451367abfdaac83a169e3e4f3a340a"}
Nov 03 10:55:18.440765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.440730    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81f79080d2e6d8729661b581d3313cb921451367abfdaac83a169e3e4f3a340a"
Nov 03 10:55:18.442595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.442546    5817 generic.go:356] "Generic (PLEG): container finished" podID="f3704fcb-71c9-4c94-bd03-a8d90dc9e79e" containerID="ffcd4080ab3296e52d6e920fe843b4180cfa5ca5f054e8c63a99af6e995b4e13" exitCode=0
Nov 03 10:55:18.442768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.442666    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg" event={"ID":"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e","Type":"ContainerDied","Data":"ffcd4080ab3296e52d6e920fe843b4180cfa5ca5f054e8c63a99af6e995b4e13"}
Nov 03 10:55:18.543682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:18.543149    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:19.074342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.074311    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"]
Nov 03 10:55:19.075183 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.075163    5817 cpu_manager.go:414] "containerMap: removing container" podUID="15465683-12e9-4bb3-9592-5e5abe524338" containerName="revoke-keys"
Nov 03 10:55:19.075266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.075185    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15465683-12e9-4bb3-9592-5e5abe524338" containerName="revoke-keys"
Nov 03 10:55:19.075266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.075199    5817 cpu_manager.go:414] "containerMap: removing container" podUID="37cfba9a-3846-47ac-8f76-536d5e5d685d" containerName="revoke-keys"
Nov 03 10:55:19.075266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.075208    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cfba9a-3846-47ac-8f76-536d5e5d685d" containerName="revoke-keys"
Nov 03 10:55:19.075372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.075323    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="15465683-12e9-4bb3-9592-5e5abe524338" containerName="revoke-keys"
Nov 03 10:55:19.075372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.075336    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cfba9a-3846-47ac-8f76-536d5e5d685d" containerName="revoke-keys"
Nov 03 10:55:19.078889 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.078872    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.085843 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.085814    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"]
Nov 03 10:55:19.130562 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.130530    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.130760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.130669    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.130760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.130751    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97r7\" (UniqueName: \"kubernetes.io/projected/85d21ae5-8da1-4b48-9280-6470e93d5c17-kube-api-access-f97r7\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.232120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.232086    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.232620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.232200    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.232620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.232333    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97r7\" (UniqueName: \"kubernetes.io/projected/85d21ae5-8da1-4b48-9280-6470e93d5c17-kube-api-access-f97r7\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.232849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.232827    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.234840 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.234814    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.241213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.241191    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97r7\" (UniqueName: \"kubernetes.io/projected/85d21ae5-8da1-4b48-9280-6470e93d5c17-kube-api-access-f97r7\") pod \"maas-api-revoke-sub-403d0a86-qz2nn\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.344299 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.344214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:19.409479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.409435    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:19.666096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.666071    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:19.702891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.702866    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"]
Nov 03 10:55:19.741400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.741368    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-kube-api-access-l22qb\" (UniqueName: \"kubernetes.io/projected/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-kube-api-access-l22qb\") pod \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") "
Nov 03 10:55:19.742049 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.742027    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-xks-ca\") pod \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") "
Nov 03 10:55:19.742348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.742331    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-openshift-ca\") pod \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\" (UID: \"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e\") "
Nov 03 10:55:19.742871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.742739    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-openshift-ca" pod "f3704fcb-71c9-4c94-bd03-a8d90dc9e79e" (UID: "f3704fcb-71c9-4c94-bd03-a8d90dc9e79e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:19.743287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.743265    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.744172 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.744153    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-kube-api-access-l22qb" pod "f3704fcb-71c9-4c94-bd03-a8d90dc9e79e" (UID: "f3704fcb-71c9-4c94-bd03-a8d90dc9e79e"). InnerVolumeSpecName "kube-api-access-l22qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:19.744336 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.744315    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-xks-ca" pod "f3704fcb-71c9-4c94-bd03-a8d90dc9e79e" (UID: "f3704fcb-71c9-4c94-bd03-a8d90dc9e79e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:19.844722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.844689    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l22qb\" (UniqueName: \"kubernetes.io/projected/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-kube-api-access-l22qb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.844722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.844719    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.963218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:19.958723    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:20.462329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:20.462296    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn" event={"ID":"85d21ae5-8da1-4b48-9280-6470e93d5c17","Type":"ContainerStarted","Data":"7bde258eab4e8692a684df792e4f81082bc0541ed0aeb51970a8290fcc5ad7d0"}
Nov 03 10:55:20.462786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:20.462351    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn" event={"ID":"85d21ae5-8da1-4b48-9280-6470e93d5c17","Type":"ContainerStarted","Data":"36be6e3a809bebea7987283f70866ed8ed6f40096e691207bf53f5a21787eafa"}
Nov 03 10:55:20.463703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:20.463681    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"
Nov 03 10:55:20.463830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:20.463715    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg" event={"ID":"f3704fcb-71c9-4c94-bd03-a8d90dc9e79e","Type":"ContainerDied","Data":"566163dbac6eb7516fc778a27de382098dd78fb5200ba52dc8ad727fcc976b88"}
Nov 03 10:55:20.463830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:20.463758    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="566163dbac6eb7516fc778a27de382098dd78fb5200ba52dc8ad727fcc976b88"
Nov 03 10:55:20.484466 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:20.484424    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn" podStartSLOduration=1.48441298 podStartE2EDuration="1.48441298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:55:20.479620176 +0000 UTC m=+1550.685202227" watchObservedRunningTime="2026-11-03 10:55:20.48441298 +0000 UTC m=+1550.689995030"
Nov 03 10:55:20.551254 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:20.551219    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:21.149077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:21.148973    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:21.473856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:21.473828    5817 generic.go:356] "Generic (PLEG): container finished" podID="85d21ae5-8da1-4b48-9280-6470e93d5c17" containerID="7bde258eab4e8692a684df792e4f81082bc0541ed0aeb51970a8290fcc5ad7d0" exitCode=0
Nov 03 10:55:21.474265 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:21.473911    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn" event={"ID":"85d21ae5-8da1-4b48-9280-6470e93d5c17","Type":"ContainerDied","Data":"7bde258eab4e8692a684df792e4f81082bc0541ed0aeb51970a8290fcc5ad7d0"}
Nov 03 10:55:22.056872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.053723    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:22.419540 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:55:22.419447    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:55:22.660360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.660336    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:22.698893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.698870    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-openshift-ca\") pod \"85d21ae5-8da1-4b48-9280-6470e93d5c17\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") "
Nov 03 10:55:22.699075 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.698915    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d21ae5-8da1-4b48-9280-6470e93d5c17-kube-api-access-f97r7\" (UniqueName: \"kubernetes.io/projected/85d21ae5-8da1-4b48-9280-6470e93d5c17-kube-api-access-f97r7\") pod \"85d21ae5-8da1-4b48-9280-6470e93d5c17\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") "
Nov 03 10:55:22.699222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.699201    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-xks-ca\") pod \"85d21ae5-8da1-4b48-9280-6470e93d5c17\" (UID: \"85d21ae5-8da1-4b48-9280-6470e93d5c17\") "
Nov 03 10:55:22.700314 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.700283    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-openshift-ca" pod "85d21ae5-8da1-4b48-9280-6470e93d5c17" (UID: "85d21ae5-8da1-4b48-9280-6470e93d5c17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:22.700862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.700841    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:22.703608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.702409    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d21ae5-8da1-4b48-9280-6470e93d5c17-kube-api-access-f97r7" pod "85d21ae5-8da1-4b48-9280-6470e93d5c17" (UID: "85d21ae5-8da1-4b48-9280-6470e93d5c17"). InnerVolumeSpecName "kube-api-access-f97r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:22.703608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.703146    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-xks-ca" pod "85d21ae5-8da1-4b48-9280-6470e93d5c17" (UID: "85d21ae5-8da1-4b48-9280-6470e93d5c17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:22.768489 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.767656    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:22.802959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.802925    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97r7\" (UniqueName: \"kubernetes.io/projected/85d21ae5-8da1-4b48-9280-6470e93d5c17-kube-api-access-f97r7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:22.802959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:22.802959    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d21ae5-8da1-4b48-9280-6470e93d5c17-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.371651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.371178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:23.494319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.494287    5817 generic.go:356] "Generic (PLEG): container finished" podID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerID="22335baf83eef623ce60c4332cf236a8bd188af309cd0de381dc0418b7600892" exitCode=0
Nov 03 10:55:23.494503 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.494355    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" event={"ID":"235f4586-fb1b-400d-9a25-fb73a5aa5630","Type":"ContainerDied","Data":"22335baf83eef623ce60c4332cf236a8bd188af309cd0de381dc0418b7600892"}
Nov 03 10:55:23.494503 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.494411    5817 scope.go:122] "RemoveContainer" containerID="adf053cb278ded75802f7c25edd396ee1024a08f3c1991b88479071bbea0d83c"
Nov 03 10:55:23.496095 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.496070    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"
Nov 03 10:55:23.496194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.496088    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn" event={"ID":"85d21ae5-8da1-4b48-9280-6470e93d5c17","Type":"ContainerDied","Data":"36be6e3a809bebea7987283f70866ed8ed6f40096e691207bf53f5a21787eafa"}
Nov 03 10:55:23.496194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.496137    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36be6e3a809bebea7987283f70866ed8ed6f40096e691207bf53f5a21787eafa"
Nov 03 10:55:23.532745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.532723    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:55:23.616090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616050    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-podinfo\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616118    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-certs\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616169    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-kube-api-access-tmp2s\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-kube-api-access-tmp2s\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616244    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-credential-socket\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616443 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616277    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-envoy\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616443 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616346    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-data\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616465    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-token\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-token\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616494    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-socket\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616643    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235f4586-fb1b-400d-9a25-fb73a5aa5630-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235f4586-fb1b-400d-9a25-fb73a5aa5630-istiod-ca-cert\") pod \"235f4586-fb1b-400d-9a25-fb73a5aa5630\" (UID: \"235f4586-fb1b-400d-9a25-fb73a5aa5630\") "
Nov 03 10:55:23.616675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616444    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-certs" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:23.616867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.616842    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-data" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:23.617101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.617058    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-credential-socket" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:23.617101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.617077    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-socket" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:23.617258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.617171    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235f4586-fb1b-400d-9a25-fb73a5aa5630-istiod-ca-cert" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:23.617436 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.617418    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235f4586-fb1b-400d-9a25-fb73a5aa5630-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.617514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.617442    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.617514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.617457    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.617514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.617470    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.617514 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.617486    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.618932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.618872    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-token" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:23.619029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.618953    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-podinfo" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:55:23.619329 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.619230    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-envoy" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:23.620587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.620550    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-kube-api-access-tmp2s" pod "235f4586-fb1b-400d-9a25-fb73a5aa5630" (UID: "235f4586-fb1b-400d-9a25-fb73a5aa5630"). InnerVolumeSpecName "kube-api-access-tmp2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:23.719121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.719089    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmp2s\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-kube-api-access-tmp2s\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.719121 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.719122    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.719628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.719137    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:23.719628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:23.719151    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235f4586-fb1b-400d-9a25-fb73a5aa5630-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:24.041211 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.041173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:24.460759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.460016    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:24.506118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.506089    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"
Nov 03 10:55:24.506322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.506078    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5" event={"ID":"235f4586-fb1b-400d-9a25-fb73a5aa5630","Type":"ContainerDied","Data":"7853902f9160b3bb567a9c01504c3d12304b1633770e9282ba0c0cfa364e60ea"}
Nov 03 10:55:24.506322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.506224    5817 scope.go:122] "RemoveContainer" containerID="22335baf83eef623ce60c4332cf236a8bd188af309cd0de381dc0418b7600892"
Nov 03 10:55:24.526231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.526202    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"]
Nov 03 10:55:24.532521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.532495    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-55gz5"]
Nov 03 10:55:24.591907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.591870    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:24.903097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:24.903018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:25.794594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:25.787215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:26.000598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:25.998223    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" path="/var/lib/kubelet/pods/235f4586-fb1b-400d-9a25-fb73a5aa5630/volumes"
Nov 03 10:55:26.660626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:26.653477    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:27.437966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:27.427558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:28.168886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:28.168839    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:28.831209 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:28.831168    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:28.977597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:28.971619    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:29.115742 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.115657    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:29.345067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.345036    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"]
Nov 03 10:55:29.349221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.349195    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-18264e-rgmdg"]
Nov 03 10:55:29.370310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.370232    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:29.511964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.511927    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:29.886801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.886763    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:29.904766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.904735    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3704fcb-71c9-4c94-bd03-a8d90dc9e79e" path="/var/lib/kubelet/pods/f3704fcb-71c9-4c94-bd03-a8d90dc9e79e/volumes"
Nov 03 10:55:29.924016 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.923992    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"]
Nov 03 10:55:29.924734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924711    5817 cpu_manager.go:414] "containerMap: removing container" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy"
Nov 03 10:55:29.924734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924733    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy"
Nov 03 10:55:29.924932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924770    5817 cpu_manager.go:414] "containerMap: removing container" podUID="85d21ae5-8da1-4b48-9280-6470e93d5c17" containerName="revoke-keys"
Nov 03 10:55:29.924932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924778    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d21ae5-8da1-4b48-9280-6470e93d5c17" containerName="revoke-keys"
Nov 03 10:55:29.924932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924823    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f3704fcb-71c9-4c94-bd03-a8d90dc9e79e" containerName="revoke-keys"
Nov 03 10:55:29.924932 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924830    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3704fcb-71c9-4c94-bd03-a8d90dc9e79e" containerName="revoke-keys"
Nov 03 10:55:29.925148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924960    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3704fcb-71c9-4c94-bd03-a8d90dc9e79e" containerName="revoke-keys"
Nov 03 10:55:29.925148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924974    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy"
Nov 03 10:55:29.925148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.924985    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy"
Nov 03 10:55:29.925148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.925000    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d21ae5-8da1-4b48-9280-6470e93d5c17" containerName="revoke-keys"
Nov 03 10:55:29.930206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.930186    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:29.936018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.934746    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:55:29.936018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.935975    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:55:29.937628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.937602    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"]
Nov 03 10:55:29.939884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.939805    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:29.939884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.939848    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:29.940074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:29.939954    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvcd\" (UniqueName: \"kubernetes.io/projected/129836b0-55cd-4d36-9dbd-3080928ec67c-kube-api-access-jlvcd\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:30.040789 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.040756    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvcd\" (UniqueName: \"kubernetes.io/projected/129836b0-55cd-4d36-9dbd-3080928ec67c-kube-api-access-jlvcd\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:30.040976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.040955    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:30.041025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.041001    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:30.041638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.041616    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:30.043769 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.043746    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:30.048549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.048528    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvcd\" (UniqueName: \"kubernetes.io/projected/129836b0-55cd-4d36-9dbd-3080928ec67c-kube-api-access-jlvcd\") pod \"maas-api-revoke-sub-e1bf6ca1-fn7s9\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:30.251037 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.251010    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:30.460767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.460557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:30.536032 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.535999    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"]
Nov 03 10:55:30.536979 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:55:30.536950    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129836b0_55cd_4d36_9dbd_3080928ec67c.slice/crio-66ab9aec85d4a4d97087e3f39e0fe70a4176568440d66874327ee8447d4468d9 WatchSource:0}: Error finding container 66ab9aec85d4a4d97087e3f39e0fe70a4176568440d66874327ee8447d4468d9: Status 404 returned error can't find the container with id 66ab9aec85d4a4d97087e3f39e0fe70a4176568440d66874327ee8447d4468d9
Nov 03 10:55:30.617213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:30.617177    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9" event={"ID":"129836b0-55cd-4d36-9dbd-3080928ec67c","Type":"ContainerStarted","Data":"66ab9aec85d4a4d97087e3f39e0fe70a4176568440d66874327ee8447d4468d9"}
Nov 03 10:55:31.169439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.169340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:31.465828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.465804    5817 scope.go:122] "RemoveContainer" containerID="14cbf2cbb95b5c0e255572eb7af8fce35e3d1528258ac7f58dc3222d0c4d4150"
Nov 03 10:55:31.494784 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.494761    5817 scope.go:122] "RemoveContainer" containerID="f5c3f2665269a9d856973a2593e248c386012077198f9dcc73f236d1930054e1"
Nov 03 10:55:31.627854 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.627826    5817 generic.go:356] "Generic (PLEG): container finished" podID="129836b0-55cd-4d36-9dbd-3080928ec67c" containerID="93ad65c5bd1f7046c80bd57a52ca7b3862d74f6b52015ad6dbd169f208f8c13e" exitCode=0
Nov 03 10:55:31.628003 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.627917    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9" event={"ID":"129836b0-55cd-4d36-9dbd-3080928ec67c","Type":"ContainerDied","Data":"93ad65c5bd1f7046c80bd57a52ca7b3862d74f6b52015ad6dbd169f208f8c13e"}
Nov 03 10:55:31.783020 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.782935    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:31.914598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.909366    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"]
Nov 03 10:55:31.914598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.913613    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:31.914598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:31.913874    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-343e30-mvmds"]
Nov 03 10:55:32.056648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:32.056088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:32.462163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:32.462131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:32.532619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:32.532586    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"]
Nov 03 10:55:32.536908 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:32.536883    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cd6d01-4csj6"]
Nov 03 10:55:32.600070 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:32.600044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:32.739199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:32.739132    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:32.985954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:32.985927    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:33.116683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.116603    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-openshift-ca\") pod \"129836b0-55cd-4d36-9dbd-3080928ec67c\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") "
Nov 03 10:55:33.116857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.116706    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129836b0-55cd-4d36-9dbd-3080928ec67c-kube-api-access-jlvcd\" (UniqueName: \"kubernetes.io/projected/129836b0-55cd-4d36-9dbd-3080928ec67c-kube-api-access-jlvcd\") pod \"129836b0-55cd-4d36-9dbd-3080928ec67c\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") "
Nov 03 10:55:33.116857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.116765    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-xks-ca\") pod \"129836b0-55cd-4d36-9dbd-3080928ec67c\" (UID: \"129836b0-55cd-4d36-9dbd-3080928ec67c\") "
Nov 03 10:55:33.116987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.116962    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-openshift-ca" pod "129836b0-55cd-4d36-9dbd-3080928ec67c" (UID: "129836b0-55cd-4d36-9dbd-3080928ec67c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:33.117541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.117515    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.119077 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.119053    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-xks-ca" pod "129836b0-55cd-4d36-9dbd-3080928ec67c" (UID: "129836b0-55cd-4d36-9dbd-3080928ec67c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:33.119190 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.119154    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129836b0-55cd-4d36-9dbd-3080928ec67c-kube-api-access-jlvcd" pod "129836b0-55cd-4d36-9dbd-3080928ec67c" (UID: "129836b0-55cd-4d36-9dbd-3080928ec67c"). InnerVolumeSpecName "kube-api-access-jlvcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:33.218410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.218378    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlvcd\" (UniqueName: \"kubernetes.io/projected/129836b0-55cd-4d36-9dbd-3080928ec67c-kube-api-access-jlvcd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.218410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.218407    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129836b0-55cd-4d36-9dbd-3080928ec67c-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.651928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.651901    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"
Nov 03 10:55:33.652312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.651945    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9" event={"ID":"129836b0-55cd-4d36-9dbd-3080928ec67c","Type":"ContainerDied","Data":"66ab9aec85d4a4d97087e3f39e0fe70a4176568440d66874327ee8447d4468d9"}
Nov 03 10:55:33.652312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.652046    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66ab9aec85d4a4d97087e3f39e0fe70a4176568440d66874327ee8447d4468d9"
Nov 03 10:55:33.681631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.680624    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:33.902643 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.902539    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257a5ec2-87c8-4eda-b090-161ab102a5ab" path="/var/lib/kubelet/pods/257a5ec2-87c8-4eda-b090-161ab102a5ab/volumes"
Nov 03 10:55:33.903046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:33.903031    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cfba9a-3846-47ac-8f76-536d5e5d685d" path="/var/lib/kubelet/pods/37cfba9a-3846-47ac-8f76-536d5e5d685d/volumes"
Nov 03 10:55:34.414972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:34.414938    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:35.055671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:35.055623    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:35.670979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:35.670934    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:36.061277 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.061248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:36.195424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.195392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:36.351344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.351223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:36.480675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.480564    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:36.889559 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.889520    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"]
Nov 03 10:55:36.890764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.890738    5817 cpu_manager.go:414] "containerMap: removing container" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy"
Nov 03 10:55:36.890764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.890763    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235f4586-fb1b-400d-9a25-fb73a5aa5630" containerName="istio-proxy"
Nov 03 10:55:36.891009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.890880    5817 cpu_manager.go:414] "containerMap: removing container" podUID="129836b0-55cd-4d36-9dbd-3080928ec67c" containerName="revoke-keys"
Nov 03 10:55:36.891009 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.890888    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129836b0-55cd-4d36-9dbd-3080928ec67c" containerName="revoke-keys"
Nov 03 10:55:36.892340 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.892314    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="129836b0-55cd-4d36-9dbd-3080928ec67c" containerName="revoke-keys"
Nov 03 10:55:36.900999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.900973    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:36.904998 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.904971    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"]
Nov 03 10:55:36.906131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.906090    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:55:36.906896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.906879    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:55:36.990104 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.990071    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:36.990288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.990269    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fcv\" (UniqueName: \"kubernetes.io/projected/27a77a8b-9e71-4c89-ae82-36170c7bea68-kube-api-access-42fcv\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:36.990803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:36.990683    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:37.091617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.091589    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:37.092086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.091630    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:37.092086 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.091707    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fcv\" (UniqueName: \"kubernetes.io/projected/27a77a8b-9e71-4c89-ae82-36170c7bea68-kube-api-access-42fcv\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:37.092253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.092235    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:37.094232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.094203    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:37.099202 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.099181    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fcv\" (UniqueName: \"kubernetes.io/projected/27a77a8b-9e71-4c89-ae82-36170c7bea68-kube-api-access-42fcv\") pod \"maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:37.219158 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.219119    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:37.465716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.465682    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:37.503411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.503383    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"]
Nov 03 10:55:37.504141 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:55:37.504107    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a77a8b_9e71_4c89_ae82_36170c7bea68.slice/crio-daeb49ae7dcac50176c944223a91b9c7179c9d1d462ffee848cfa5690f0a6d18 WatchSource:0}: Error finding container daeb49ae7dcac50176c944223a91b9c7179c9d1d462ffee848cfa5690f0a6d18: Status 404 returned error can't find the container with id daeb49ae7dcac50176c944223a91b9c7179c9d1d462ffee848cfa5690f0a6d18
Nov 03 10:55:37.695538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:37.695506    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq" event={"ID":"27a77a8b-9e71-4c89-ae82-36170c7bea68","Type":"ContainerStarted","Data":"daeb49ae7dcac50176c944223a91b9c7179c9d1d462ffee848cfa5690f0a6d18"}
Nov 03 10:55:38.168601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:38.168495    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:38.476043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:38.476006    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:38.617368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:38.617326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:38.711954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:38.711503    5817 generic.go:356] "Generic (PLEG): container finished" podID="27a77a8b-9e71-4c89-ae82-36170c7bea68" containerID="c9be68fa9c5e6db92cab48a4024b722c7e31015f30899e002b67e2f891c23adf" exitCode=0
Nov 03 10:55:38.711954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:38.711546    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq" event={"ID":"27a77a8b-9e71-4c89-ae82-36170c7bea68","Type":"ContainerDied","Data":"c9be68fa9c5e6db92cab48a4024b722c7e31015f30899e002b67e2f891c23adf"}
Nov 03 10:55:38.819950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:38.819875    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:39.166553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.165659    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:39.881090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.881067    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:39.936884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.936851    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:39.947467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.947438    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-xks-ca\") pod \"27a77a8b-9e71-4c89-ae82-36170c7bea68\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") "
Nov 03 10:55:39.947822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.947796    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a77a8b-9e71-4c89-ae82-36170c7bea68-kube-api-access-42fcv\" (UniqueName: \"kubernetes.io/projected/27a77a8b-9e71-4c89-ae82-36170c7bea68-kube-api-access-42fcv\") pod \"27a77a8b-9e71-4c89-ae82-36170c7bea68\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") "
Nov 03 10:55:39.948027 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.948007    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-openshift-ca\") pod \"27a77a8b-9e71-4c89-ae82-36170c7bea68\" (UID: \"27a77a8b-9e71-4c89-ae82-36170c7bea68\") "
Nov 03 10:55:39.949026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.948943    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-openshift-ca" pod "27a77a8b-9e71-4c89-ae82-36170c7bea68" (UID: "27a77a8b-9e71-4c89-ae82-36170c7bea68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:39.954208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.952064    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a77a8b-9e71-4c89-ae82-36170c7bea68-kube-api-access-42fcv" pod "27a77a8b-9e71-4c89-ae82-36170c7bea68" (UID: "27a77a8b-9e71-4c89-ae82-36170c7bea68"). InnerVolumeSpecName "kube-api-access-42fcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:39.954208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:39.952236    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-xks-ca" pod "27a77a8b-9e71-4c89-ae82-36170c7bea68" (UID: "27a77a8b-9e71-4c89-ae82-36170c7bea68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:40.050274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:40.050194    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:40.050274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:40.050219    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42fcv\" (UniqueName: \"kubernetes.io/projected/27a77a8b-9e71-4c89-ae82-36170c7bea68-kube-api-access-42fcv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:40.050274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:40.050232    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a77a8b-9e71-4c89-ae82-36170c7bea68-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:40.743247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:40.743219    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"
Nov 03 10:55:40.743452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:40.743246    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq" event={"ID":"27a77a8b-9e71-4c89-ae82-36170c7bea68","Type":"ContainerDied","Data":"daeb49ae7dcac50176c944223a91b9c7179c9d1d462ffee848cfa5690f0a6d18"}
Nov 03 10:55:40.743452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:40.743301    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daeb49ae7dcac50176c944223a91b9c7179c9d1d462ffee848cfa5690f0a6d18"
Nov 03 10:55:40.763440 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:40.763399    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:41.192806 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.192769    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"]
Nov 03 10:55:41.193344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.193329    5817 cpu_manager.go:414] "containerMap: removing container" podUID="27a77a8b-9e71-4c89-ae82-36170c7bea68" containerName="revoke-keys"
Nov 03 10:55:41.193344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.193343    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a77a8b-9e71-4c89-ae82-36170c7bea68" containerName="revoke-keys"
Nov 03 10:55:41.193467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.193447    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a77a8b-9e71-4c89-ae82-36170c7bea68" containerName="revoke-keys"
Nov 03 10:55:41.196970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.196950    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.200912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.200890    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:55:41.201543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.201520    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:55:41.205429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.205399    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"]
Nov 03 10:55:41.306761 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.306735    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.306903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.306780    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjqg\" (UniqueName: \"kubernetes.io/projected/2c6621ad-9182-4645-9119-2b7e7c398ff0-kube-api-access-hfjqg\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.306903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.306877    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.408310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.408279    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.408461 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.408328    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjqg\" (UniqueName: \"kubernetes.io/projected/2c6621ad-9182-4645-9119-2b7e7c398ff0-kube-api-access-hfjqg\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.408461 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.408361    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.409069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.409039    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.410999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.410974    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.416828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.416800    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjqg\" (UniqueName: \"kubernetes.io/projected/2c6621ad-9182-4645-9119-2b7e7c398ff0-kube-api-access-hfjqg\") pod \"maas-api-revoke-sub-c151691e-5lgdd\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.439057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.439024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:41.520675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.520603    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:41.803615 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:55:41.803116    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6621ad_9182_4645_9119_2b7e7c398ff0.slice/crio-7e77c317f4e8b4dd33b6e24a692bca4974c872e626bb9f743795fdbf56988bc3 WatchSource:0}: Error finding container 7e77c317f4e8b4dd33b6e24a692bca4974c872e626bb9f743795fdbf56988bc3: Status 404 returned error can't find the container with id 7e77c317f4e8b4dd33b6e24a692bca4974c872e626bb9f743795fdbf56988bc3
Nov 03 10:55:41.805315 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:41.805292    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"]
Nov 03 10:55:42.067668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:42.066712    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:42.613905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:42.613873    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:42.776549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:42.776513    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:42.781644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:42.781613    5817 generic.go:356] "Generic (PLEG): container finished" podID="2c6621ad-9182-4645-9119-2b7e7c398ff0" containerID="b40b9274ebf454fdbde84b7ba6e60fc62eb5b3fc7532b2ca3d18faa08e3d2bdb" exitCode=0
Nov 03 10:55:42.781849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:42.781751    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd" event={"ID":"2c6621ad-9182-4645-9119-2b7e7c398ff0","Type":"ContainerDied","Data":"b40b9274ebf454fdbde84b7ba6e60fc62eb5b3fc7532b2ca3d18faa08e3d2bdb"}
Nov 03 10:55:42.781849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:42.781805    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd" event={"ID":"2c6621ad-9182-4645-9119-2b7e7c398ff0","Type":"ContainerStarted","Data":"7e77c317f4e8b4dd33b6e24a692bca4974c872e626bb9f743795fdbf56988bc3"}
Nov 03 10:55:43.075540 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:43.075507    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:43.548770 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:43.548731    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:43.961934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:43.961911    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:44.047063 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.047037    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-openshift-ca\") pod \"2c6621ad-9182-4645-9119-2b7e7c398ff0\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") "
Nov 03 10:55:44.047230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.047105    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-xks-ca\") pod \"2c6621ad-9182-4645-9119-2b7e7c398ff0\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") "
Nov 03 10:55:44.047230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.047166    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6621ad-9182-4645-9119-2b7e7c398ff0-kube-api-access-hfjqg\" (UniqueName: \"kubernetes.io/projected/2c6621ad-9182-4645-9119-2b7e7c398ff0-kube-api-access-hfjqg\") pod \"2c6621ad-9182-4645-9119-2b7e7c398ff0\" (UID: \"2c6621ad-9182-4645-9119-2b7e7c398ff0\") "
Nov 03 10:55:44.047591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.047433    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-openshift-ca" pod "2c6621ad-9182-4645-9119-2b7e7c398ff0" (UID: "2c6621ad-9182-4645-9119-2b7e7c398ff0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:44.047929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.047903    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.052616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.049922    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-xks-ca" pod "2c6621ad-9182-4645-9119-2b7e7c398ff0" (UID: "2c6621ad-9182-4645-9119-2b7e7c398ff0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:44.052616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.050026    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6621ad-9182-4645-9119-2b7e7c398ff0-kube-api-access-hfjqg" pod "2c6621ad-9182-4645-9119-2b7e7c398ff0" (UID: "2c6621ad-9182-4645-9119-2b7e7c398ff0"). InnerVolumeSpecName "kube-api-access-hfjqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:44.148987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.148914    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6621ad-9182-4645-9119-2b7e7c398ff0-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.148987 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.148943    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjqg\" (UniqueName: \"kubernetes.io/projected/2c6621ad-9182-4645-9119-2b7e7c398ff0-kube-api-access-hfjqg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.244386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.244050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:44.801741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.801694    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd" event={"ID":"2c6621ad-9182-4645-9119-2b7e7c398ff0","Type":"ContainerDied","Data":"7e77c317f4e8b4dd33b6e24a692bca4974c872e626bb9f743795fdbf56988bc3"}
Nov 03 10:55:44.801931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.801760    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e77c317f4e8b4dd33b6e24a692bca4974c872e626bb9f743795fdbf56988bc3"
Nov 03 10:55:44.801931 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.801706    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"
Nov 03 10:55:44.877817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:44.877783    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:45.435271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:45.434854    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:45.555191 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:45.555154    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:55:46.152501 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:46.152398    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:47.226690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:47.223727    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:47.381174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:47.381141    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:47.797894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:47.797852    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:47.836768 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:47.836726    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.056770 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.056668    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.063673 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.063639    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.125596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.119656    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:48.154812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.154761    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.182390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.182345    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.322656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.303889    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.571599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.562657    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.581650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.578647    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.762080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.762034    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.798706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.798659    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.799501 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.799475    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:55:48.835298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.835221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:48.837691 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.837659    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 10:55:48.838215 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:48.838196    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:55:49.244865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.044551    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.244865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.057073    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.244865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.057547    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:55:49.244865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.162926    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.244865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.198778    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded"
Nov 03 10:55:49.244865 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.199017    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:55:49.535754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.535659    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.570358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.553793    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:55:49.740594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.584745    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:55:49.740594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.689225    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:55:49.740594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.655854    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.740594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.686320    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:55:49.740594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.699020    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:49.806645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.806017    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 10:55:49.843595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.833187    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:55:49.843595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.835199    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 10:55:49.868599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.853280    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.940159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:49.940129    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:50.068038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.061838    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:55:50.068038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.061889    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:55:50.068038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.061923    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:55:50.068038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.061953    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:55:50.068038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.061983    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:55:50.068038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.062010    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:55:50.068038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.062038    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:55:50.082995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.082964    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:50.208170 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.208137    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"]
Nov 03 10:55:50.215324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.215297    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8bbb6e-99wfq"]
Nov 03 10:55:50.229231 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.229198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:50.412651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.402009    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:50.699375 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:50.699343    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:51.180914 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:51.180880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:51.866267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:51.866235    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:51.905392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:51.905360    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a77a8b-9e71-4c89-ae82-36170c7bea68" path="/var/lib/kubelet/pods/27a77a8b-9e71-4c89-ae82-36170c7bea68/volumes"
Nov 03 10:55:52.412992 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:55:52.412952    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:55:52.747624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:52.746023    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:53.448911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.448877    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:53.561351 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.561318    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"]
Nov 03 10:55:53.561966 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.561951    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2c6621ad-9182-4645-9119-2b7e7c398ff0" containerName="revoke-keys"
Nov 03 10:55:53.562022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.561969    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6621ad-9182-4645-9119-2b7e7c398ff0" containerName="revoke-keys"
Nov 03 10:55:53.562078 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.562069    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6621ad-9182-4645-9119-2b7e7c398ff0" containerName="revoke-keys"
Nov 03 10:55:53.565596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.565566    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.569766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.569263    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:55:53.569766 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.569456    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:55:53.573244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.573221    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"]
Nov 03 10:55:53.586494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.586464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:53.603860 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.603827    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff819c8-689a-4878-b560-9da617880b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.604063 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.604041    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpnr\" (UniqueName: \"kubernetes.io/projected/aff819c8-689a-4878-b560-9da617880b3f-kube-api-access-fvpnr\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.604167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.604152    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff819c8-689a-4878-b560-9da617880b3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.705634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.705489    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff819c8-689a-4878-b560-9da617880b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.705833 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.705684    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpnr\" (UniqueName: \"kubernetes.io/projected/aff819c8-689a-4878-b560-9da617880b3f-kube-api-access-fvpnr\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.705833 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.705783    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff819c8-689a-4878-b560-9da617880b3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.706300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.706273    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff819c8-689a-4878-b560-9da617880b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.708542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.708516    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff819c8-689a-4878-b560-9da617880b3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.718370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.718340    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpnr\" (UniqueName: \"kubernetes.io/projected/aff819c8-689a-4878-b560-9da617880b3f-kube-api-access-fvpnr\") pod \"maas-api-revoke-sub-6532dd40-b7f8g\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:53.881412 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:53.881383    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:54.181759 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:55:54.181716    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff819c8_689a_4878_b560_9da617880b3f.slice/crio-eb1168663606dd853e756578515322d015713ca5a85bf3261c11257962b2cc4a WatchSource:0}: Error finding container eb1168663606dd853e756578515322d015713ca5a85bf3261c11257962b2cc4a: Status 404 returned error can't find the container with id eb1168663606dd853e756578515322d015713ca5a85bf3261c11257962b2cc4a
Nov 03 10:55:54.182017 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:54.181995    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"]
Nov 03 10:55:54.302594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:54.298953    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:54.463682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:54.463646    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g" event={"ID":"aff819c8-689a-4878-b560-9da617880b3f","Type":"ContainerStarted","Data":"eb1168663606dd853e756578515322d015713ca5a85bf3261c11257962b2cc4a"}
Nov 03 10:55:55.052300 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:55.052268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:55.473287 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:55.473256    5817 generic.go:356] "Generic (PLEG): container finished" podID="aff819c8-689a-4878-b560-9da617880b3f" containerID="f9ef28ac2562769a5b20c8712fa178ca778818f567b74c8ea65489f6718a265b" exitCode=0
Nov 03 10:55:55.473717 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:55.473346    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g" event={"ID":"aff819c8-689a-4878-b560-9da617880b3f","Type":"ContainerDied","Data":"f9ef28ac2562769a5b20c8712fa178ca778818f567b74c8ea65489f6718a265b"}
Nov 03 10:55:55.705658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:55.705625    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:56.255056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.255018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:56.469601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.469535    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:56.632412 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.632377    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:56.658515 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.658492    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:56.764058 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.764024    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff819c8-689a-4878-b560-9da617880b3f-kube-api-access-fvpnr\" (UniqueName: \"kubernetes.io/projected/aff819c8-689a-4878-b560-9da617880b3f-kube-api-access-fvpnr\") pod \"aff819c8-689a-4878-b560-9da617880b3f\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") "
Nov 03 10:55:56.764263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.764187    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff819c8-689a-4878-b560-9da617880b3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff819c8-689a-4878-b560-9da617880b3f-maas-api-xks-ca\") pod \"aff819c8-689a-4878-b560-9da617880b3f\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") "
Nov 03 10:55:56.764263 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.764237    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff819c8-689a-4878-b560-9da617880b3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff819c8-689a-4878-b560-9da617880b3f-maas-api-openshift-ca\") pod \"aff819c8-689a-4878-b560-9da617880b3f\" (UID: \"aff819c8-689a-4878-b560-9da617880b3f\") "
Nov 03 10:55:56.765344 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.765314    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff819c8-689a-4878-b560-9da617880b3f-maas-api-openshift-ca" pod "aff819c8-689a-4878-b560-9da617880b3f" (UID: "aff819c8-689a-4878-b560-9da617880b3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:56.767116 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.767088    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff819c8-689a-4878-b560-9da617880b3f-kube-api-access-fvpnr" pod "aff819c8-689a-4878-b560-9da617880b3f" (UID: "aff819c8-689a-4878-b560-9da617880b3f"). InnerVolumeSpecName "kube-api-access-fvpnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:56.768665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.768624    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff819c8-689a-4878-b560-9da617880b3f-maas-api-xks-ca" pod "aff819c8-689a-4878-b560-9da617880b3f" (UID: "aff819c8-689a-4878-b560-9da617880b3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:56.856648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.856483    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:56.885603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.879324    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff819c8-689a-4878-b560-9da617880b3f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:56.885603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.879387    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff819c8-689a-4878-b560-9da617880b3f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:56.885603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:56.879407    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvpnr\" (UniqueName: \"kubernetes.io/projected/aff819c8-689a-4878-b560-9da617880b3f-kube-api-access-fvpnr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:57.009456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:57.009424    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:57.497030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:57.496995    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g" event={"ID":"aff819c8-689a-4878-b560-9da617880b3f","Type":"ContainerDied","Data":"eb1168663606dd853e756578515322d015713ca5a85bf3261c11257962b2cc4a"}
Nov 03 10:55:57.497030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:57.497011    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"
Nov 03 10:55:57.497256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:57.497040    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb1168663606dd853e756578515322d015713ca5a85bf3261c11257962b2cc4a"
Nov 03 10:55:57.704801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:57.704771    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:58.347496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.347463    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:58.479131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.479099    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"]
Nov 03 10:55:58.479379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.479354    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy" containerID="cri-o://aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b" gracePeriod=30
Nov 03 10:55:58.568727 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.568690    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:58.849713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.849680    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"]
Nov 03 10:55:58.850550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.850524    5817 cpu_manager.go:414] "containerMap: removing container" podUID="aff819c8-689a-4878-b560-9da617880b3f" containerName="revoke-keys"
Nov 03 10:55:58.850550 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.850549    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff819c8-689a-4878-b560-9da617880b3f" containerName="revoke-keys"
Nov 03 10:55:58.850791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.850734    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff819c8-689a-4878-b560-9da617880b3f" containerName="revoke-keys"
Nov 03 10:55:58.854108 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.854085    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:58.864356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.864332    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e92a08-openshift-default-dockercfg-j59b6\""
Nov 03 10:55:58.868715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.868690    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"]
Nov 03 10:55:58.943563 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:58.943269    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:59.026370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026341    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-envoy\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.026518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026412    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.026518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026491    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e1506-6c9f-4227-8f26-567737295e48-istio-podinfo\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.026631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026536    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-data\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.026671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026629    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-certs\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.026746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026677    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-credential-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.026746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026702    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hjw\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-kube-api-access-85hjw\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.026861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026794    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-istio-token\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.026945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.026930    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e1506-6c9f-4227-8f26-567737295e48-istiod-ca-cert\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.090593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.089985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:55:59.138648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.137467    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e1506-6c9f-4227-8f26-567737295e48-istio-podinfo\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.138648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.137556    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-data\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.138648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.137611    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-certs\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.138946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.137640    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-credential-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.138946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.138918    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hjw\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-kube-api-access-85hjw\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.139068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.138981    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-istio-token\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.139068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.139058    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e1506-6c9f-4227-8f26-567737295e48-istiod-ca-cert\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.139257 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.139234    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-envoy\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.139357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.139335    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.139796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.139768    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.139923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.138043    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-data\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.139923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.138783    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-credential-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.139923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.138535    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-certs\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.140821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.140793    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e1506-6c9f-4227-8f26-567737295e48-istiod-ca-cert\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.169652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.163469    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-istio-token\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.170890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.170842    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-envoy\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.176122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.176092    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e1506-6c9f-4227-8f26-567737295e48-istio-podinfo\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.180985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.179941    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hjw\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-kube-api-access-85hjw\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.180985 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.180186    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:55:59.532801 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:55:59.532756    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061e1506_6c9f_4227_8f26_567737295e48.slice/crio-1072f94b7715b6ffe848cd2d860ee1e9685417a05967e6321a78ca5ab0c70cf8 WatchSource:0}: Error finding container 1072f94b7715b6ffe848cd2d860ee1e9685417a05967e6321a78ca5ab0c70cf8: Status 404 returned error can't find the container with id 1072f94b7715b6ffe848cd2d860ee1e9685417a05967e6321a78ca5ab0c70cf8
Nov 03 10:55:59.533526 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.533493    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"]
Nov 03 10:55:59.535500 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.535464    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:55:59.535585 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.535549    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:55:59.535628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:55:59.535615    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:00.102065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:00.102030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:00.531539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:00.531491    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" event={"ID":"061e1506-6c9f-4227-8f26-567737295e48","Type":"ContainerStarted","Data":"a75baeef46cac0b4aa16d284aa3713cae682bcfef6a615a3ba6d55c6b89a7743"}
Nov 03 10:56:00.531736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:00.531550    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" event={"ID":"061e1506-6c9f-4227-8f26-567737295e48","Type":"ContainerStarted","Data":"1072f94b7715b6ffe848cd2d860ee1e9685417a05967e6321a78ca5ab0c70cf8"}
Nov 03 10:56:00.554775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:00.554727    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podStartSLOduration=2.554712452 podStartE2EDuration="2.554712452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:00.551015518 +0000 UTC m=+1590.756597570" watchObservedRunningTime="2026-11-03 10:56:00.554712452 +0000 UTC m=+1590.760294567"
Nov 03 10:56:00.912403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:00.912170    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:01.180951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:01.180929    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:56:01.186655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:01.186629    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:56:01.548065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:01.544247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:01.559603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:01.558922    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:56:01.559778 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:01.559738    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:56:02.333295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:02.333253    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:02.458828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:02.458794    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:02.795916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:02.795877    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:02.936471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:02.936440    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:03.166424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.166337    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:03.770776 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.770701    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:56:03.823758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.823719    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istiod-ca-cert\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.823921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.823829    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-envoy\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.823921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.823910    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-certs\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.824069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.823997    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-credential-socket\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.824130 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.824068    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-podinfo\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.824130 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.824102    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-token\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-token\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.824232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.824144    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-data\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.824232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.824220    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-kube-api-access-cd85m\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-kube-api-access-cd85m\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.824345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.824253    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-socket\") pod \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\" (UID: \"69bfa55c-c68a-4b70-a943-8bbd93b6fda2\") "
Nov 03 10:56:03.825188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.825159    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-socket" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:03.825521 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.825497    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istiod-ca-cert" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:03.828254 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.828229    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-envoy" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:03.828637 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.828617    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-certs" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:03.828928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.828910    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-credential-socket" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:03.831109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.831089    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-data" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:03.831319 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.831293    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-podinfo" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:56:03.835767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.835742    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-kube-api-access-cd85m" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "kube-api-access-cd85m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:03.835949 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.835855    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-token" pod "69bfa55c-c68a-4b70-a943-8bbd93b6fda2" (UID: "69bfa55c-c68a-4b70-a943-8bbd93b6fda2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:03.845587 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.841938    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:03.935929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.935894    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:03.936258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.935941    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:03.936258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.935957    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:03.936258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.935974    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:03.936258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.935997    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:03.936258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.936017    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:03.936258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.936031    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:03.936258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.936047    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd85m\" (UniqueName: \"kubernetes.io/projected/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-kube-api-access-cd85m\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:03.936258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:03.936063    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bfa55c-c68a-4b70-a943-8bbd93b6fda2-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:04.000584 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.000536    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:04.143082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.142997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:04.582868 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.582836    5817 generic.go:356] "Generic (PLEG): container finished" podID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerID="aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b" exitCode=0
Nov 03 10:56:04.583045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.582901    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"
Nov 03 10:56:04.583045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.582912    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" event={"ID":"69bfa55c-c68a-4b70-a943-8bbd93b6fda2","Type":"ContainerDied","Data":"aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b"}
Nov 03 10:56:04.583045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.582994    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp" event={"ID":"69bfa55c-c68a-4b70-a943-8bbd93b6fda2","Type":"ContainerDied","Data":"4cdb218c7bb3f2c7f5bb2faf32ece2ed91e2a5fce8968e78ee8022b50b884475"}
Nov 03 10:56:04.583045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.583017    5817 scope.go:122] "RemoveContainer" containerID="aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b"
Nov 03 10:56:04.596978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.596961    5817 scope.go:122] "RemoveContainer" containerID="d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7"
Nov 03 10:56:04.611870 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.611138    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"]
Nov 03 10:56:04.615025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.614475    5817 scope.go:122] "RemoveContainer" containerID="aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b"
Nov 03 10:56:04.615414 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:56:04.615393    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b\": container with ID starting with aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b not found: ID does not exist" containerID="aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b"
Nov 03 10:56:04.615512 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.615421    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b"} err="failed to get container status \"aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b\": rpc error: code = NotFound desc = could not find container \"aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b\": container with ID starting with aa5c7dcc61b19e96cc21a137b27b5ae312c4eab8210e8d17c28daa958cffa52b not found: ID does not exist"
Nov 03 10:56:04.615512 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.615437    5817 scope.go:122] "RemoveContainer" containerID="d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7"
Nov 03 10:56:04.615782 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:56:04.615763    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7\": container with ID starting with d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7 not found: ID does not exist" containerID="d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7"
Nov 03 10:56:04.615846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.615787    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7"} err="failed to get container status \"d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7\": rpc error: code = NotFound desc = could not find container \"d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7\": container with ID starting with d3589f20d2d99bc2bc1ebbf7f2ca6ed7385d8704677da4766bf278000cb2bdf7 not found: ID does not exist"
Nov 03 10:56:04.621266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:04.621243    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8g4cp"]
Nov 03 10:56:05.038627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:05.028226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:05.761623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:05.761589    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:05.948596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:05.948255    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" path="/var/lib/kubelet/pods/69bfa55c-c68a-4b70-a943-8bbd93b6fda2/volumes"
Nov 03 10:56:06.466242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:06.464954    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:07.393599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:07.387601    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:08.063901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:08.063867    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:08.194075 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:08.194043    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:08.391269 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:08.391176    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:08.602594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:08.602548    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:08.939431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:08.939399    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:09.109483 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:09.109451    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:09.247870 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:09.247767    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:09.776667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:09.776606    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:10.453356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:10.452955    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:11.131942 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.131506    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:11.198724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.198682    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"]
Nov 03 10:56:11.199247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.199232    5817 cpu_manager.go:414] "containerMap: removing container" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy"
Nov 03 10:56:11.199247 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.199247    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy"
Nov 03 10:56:11.199396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.199261    5817 cpu_manager.go:414] "containerMap: removing container" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy"
Nov 03 10:56:11.199396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.199267    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy"
Nov 03 10:56:11.199396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.199373    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy"
Nov 03 10:56:11.199396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.199383    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="69bfa55c-c68a-4b70-a943-8bbd93b6fda2" containerName="istio-proxy"
Nov 03 10:56:11.204674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.204651    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.210963 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.210940    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"]
Nov 03 10:56:11.243659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.243631    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:56:11.243785 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.243766    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:56:11.290475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.290448    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8w2\" (UniqueName: \"kubernetes.io/projected/4aebc73b-ee48-41a3-8989-351f6bac0f5d-kube-api-access-zs8w2\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.290752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.290728    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.290822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.290777    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.391749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.391677    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.391907 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.391766    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8w2\" (UniqueName: \"kubernetes.io/projected/4aebc73b-ee48-41a3-8989-351f6bac0f5d-kube-api-access-zs8w2\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.391996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.391958    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.392740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.392716    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.394248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.394232    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.400028 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.399997    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8w2\" (UniqueName: \"kubernetes.io/projected/4aebc73b-ee48-41a3-8989-351f6bac0f5d-kube-api-access-zs8w2\") pod \"maas-api-revoke-sub-7ed60996-hllmp\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.524610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.524566    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:11.805736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:56:11.805703    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aebc73b_ee48_41a3_8989_351f6bac0f5d.slice/crio-34128d1f909a2002c0c93b3e4eef9eacbbca3441649457fd899f81aa625788d0 WatchSource:0}: Error finding container 34128d1f909a2002c0c93b3e4eef9eacbbca3441649457fd899f81aa625788d0: Status 404 returned error can't find the container with id 34128d1f909a2002c0c93b3e4eef9eacbbca3441649457fd899f81aa625788d0
Nov 03 10:56:11.807936 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:11.807911    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"]
Nov 03 10:56:12.033885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:12.033853    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:12.566484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:12.566427    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:12.732600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:12.732375    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:12.830727 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:12.830497    5817 generic.go:356] "Generic (PLEG): container finished" podID="4aebc73b-ee48-41a3-8989-351f6bac0f5d" containerID="e64c890afaa269f42f31184a6e3f967631374abfb98359db850628b6a54d4d2c" exitCode=0
Nov 03 10:56:12.830945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:12.830734    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp" event={"ID":"4aebc73b-ee48-41a3-8989-351f6bac0f5d","Type":"ContainerDied","Data":"e64c890afaa269f42f31184a6e3f967631374abfb98359db850628b6a54d4d2c"}
Nov 03 10:56:12.830945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:12.830826    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp" event={"ID":"4aebc73b-ee48-41a3-8989-351f6bac0f5d","Type":"ContainerStarted","Data":"34128d1f909a2002c0c93b3e4eef9eacbbca3441649457fd899f81aa625788d0"}
Nov 03 10:56:12.960951 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:12.960920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:13.627253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:13.627219    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:13.827991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:13.827952    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"]
Nov 03 10:56:13.832694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:13.832669    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:13.837612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:13.837591    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-6012b5\""
Nov 03 10:56:13.842977 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:13.842953    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"]
Nov 03 10:56:13.848878 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:13.848538    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:13.964745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:13.953445    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jh7\" (UniqueName: \"kubernetes.io/projected/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-kube-api-access-48jh7\") pod \"payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp\" (UID: \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:14.005339 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.005304    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp\" (UID: \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:14.049767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.014836    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:14.137595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.129303    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jh7\" (UniqueName: \"kubernetes.io/projected/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-kube-api-access-48jh7\") pod \"payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp\" (UID: \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:14.137595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.129499    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp\" (UID: \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:14.147653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.145691    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp\" (UID: \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:14.168642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.168449    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jh7\" (UniqueName: \"kubernetes.io/projected/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-kube-api-access-48jh7\") pod \"payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp\" (UID: \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:14.203100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.203077    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:14.230220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.230142    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-openshift-ca\") pod \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") "
Nov 03 10:56:14.230220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.230194    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aebc73b-ee48-41a3-8989-351f6bac0f5d-kube-api-access-zs8w2\" (UniqueName: \"kubernetes.io/projected/4aebc73b-ee48-41a3-8989-351f6bac0f5d-kube-api-access-zs8w2\") pod \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") "
Nov 03 10:56:14.230410 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.230265    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-xks-ca\") pod \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\" (UID: \"4aebc73b-ee48-41a3-8989-351f6bac0f5d\") "
Nov 03 10:56:14.230829 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.230804    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-openshift-ca" pod "4aebc73b-ee48-41a3-8989-351f6bac0f5d" (UID: "4aebc73b-ee48-41a3-8989-351f6bac0f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:14.230967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.230950    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:14.232755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.232729    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-xks-ca" pod "4aebc73b-ee48-41a3-8989-351f6bac0f5d" (UID: "4aebc73b-ee48-41a3-8989-351f6bac0f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:56:14.233380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.233357    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aebc73b-ee48-41a3-8989-351f6bac0f5d-kube-api-access-zs8w2" pod "4aebc73b-ee48-41a3-8989-351f6bac0f5d" (UID: "4aebc73b-ee48-41a3-8989-351f6bac0f5d"). InnerVolumeSpecName "kube-api-access-zs8w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:14.332079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.332041    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs8w2\" (UniqueName: \"kubernetes.io/projected/4aebc73b-ee48-41a3-8989-351f6bac0f5d-kube-api-access-zs8w2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:14.332079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.332081    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aebc73b-ee48-41a3-8989-351f6bac0f5d-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:14.462135 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.462107    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:14.574945 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.572850    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:14.748985 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:56:14.748954    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd47ba3_fe9f_4bc1_b4a0_959bd05ec9e1.slice/crio-b5e4bc283535b11c5a36d543751d691125f33d3e158b1951ae09b49f6a778226 WatchSource:0}: Error finding container b5e4bc283535b11c5a36d543751d691125f33d3e158b1951ae09b49f6a778226: Status 404 returned error can't find the container with id b5e4bc283535b11c5a36d543751d691125f33d3e158b1951ae09b49f6a778226
Nov 03 10:56:14.749316 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.749248    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"]
Nov 03 10:56:14.856353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.856322    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp" event={"ID":"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1","Type":"ContainerStarted","Data":"a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513"}
Nov 03 10:56:14.856516 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.856377    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp" event={"ID":"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1","Type":"ContainerStarted","Data":"b5e4bc283535b11c5a36d543751d691125f33d3e158b1951ae09b49f6a778226"}
Nov 03 10:56:14.856516 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.856416    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:14.858017 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.857994    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp" event={"ID":"4aebc73b-ee48-41a3-8989-351f6bac0f5d","Type":"ContainerDied","Data":"34128d1f909a2002c0c93b3e4eef9eacbbca3441649457fd899f81aa625788d0"}
Nov 03 10:56:14.858127 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.858017    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"
Nov 03 10:56:14.858127 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.858036    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34128d1f909a2002c0c93b3e4eef9eacbbca3441649457fd899f81aa625788d0"
Nov 03 10:56:14.873996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:14.873958    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp" podStartSLOduration=1.8739448429999999 podStartE2EDuration="1.873944843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:14.872214669 +0000 UTC m=+1605.077796717" watchObservedRunningTime="2026-11-03 10:56:14.873944843 +0000 UTC m=+1605.079526913"
Nov 03 10:56:15.251597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:15.232246    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:15.854204 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:15.854167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:16.447222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:16.444410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:17.080603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:17.079325    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:17.781429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:17.781392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:17.912497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:17.912464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:18.047142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.047039    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:18.076156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.076126    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"]
Nov 03 10:56:18.076736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.076722    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4aebc73b-ee48-41a3-8989-351f6bac0f5d" containerName="revoke-keys"
Nov 03 10:56:18.076783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.076737    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aebc73b-ee48-41a3-8989-351f6bac0f5d" containerName="revoke-keys"
Nov 03 10:56:18.076844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.076836    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aebc73b-ee48-41a3-8989-351f6bac0f5d" containerName="revoke-keys"
Nov 03 10:56:18.080391 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.080376    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.083806 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.083776    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:56:18.084206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.083864    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:56:18.088194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.088170    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"]
Nov 03 10:56:18.129557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.129524    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.129773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.129739    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvgg\" (UniqueName: \"kubernetes.io/projected/c14091ff-a442-44ec-a0f5-00cc4ca4f113-kube-api-access-grvgg\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.129849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.129819    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.233443 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.233411    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.233660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.233640    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.233837 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.233821    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvgg\" (UniqueName: \"kubernetes.io/projected/c14091ff-a442-44ec-a0f5-00cc4ca4f113-kube-api-access-grvgg\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.234502 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.234477    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.241917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.236112    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.242656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.242165    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvgg\" (UniqueName: \"kubernetes.io/projected/c14091ff-a442-44ec-a0f5-00cc4ca4f113-kube-api-access-grvgg\") pod \"maas-api-revoke-sub-cd31fbcf-psn9m\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.318991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.318911    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"]
Nov 03 10:56:18.319173 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.319149    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy" containerID="cri-o://8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a" gracePeriod=30
Nov 03 10:56:18.401259 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.401228    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:18.656223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.656121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:18.704298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.704256    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"]
Nov 03 10:56:18.931603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:18.931531    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m" event={"ID":"c14091ff-a442-44ec-a0f5-00cc4ca4f113","Type":"ContainerStarted","Data":"a5b148d65f4946072db2df69daa7690e7a8e84878c081d5887eec90b5fea0b07"}
Nov 03 10:56:19.266197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:19.266084    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:19.940796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:19.940756    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:19.943064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:19.943032    5817 generic.go:356] "Generic (PLEG): container finished" podID="c14091ff-a442-44ec-a0f5-00cc4ca4f113" containerID="d248f6500751e4cdbb46e83099a75947618d3feba71a0ecbad4a5ea3c70324e9" exitCode=0
Nov 03 10:56:19.943182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:19.943107    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m" event={"ID":"c14091ff-a442-44ec-a0f5-00cc4ca4f113","Type":"ContainerDied","Data":"d248f6500751e4cdbb46e83099a75947618d3feba71a0ecbad4a5ea3c70324e9"}
Nov 03 10:56:20.640978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:20.640945    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:21.191360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.191330    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:21.285273 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.285174    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-openshift-ca\") pod \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") "
Nov 03 10:56:21.285448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.285337    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14091ff-a442-44ec-a0f5-00cc4ca4f113-kube-api-access-grvgg\" (UniqueName: \"kubernetes.io/projected/c14091ff-a442-44ec-a0f5-00cc4ca4f113-kube-api-access-grvgg\") pod \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") "
Nov 03 10:56:21.285527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.285511    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-xks-ca\") pod \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\" (UID: \"c14091ff-a442-44ec-a0f5-00cc4ca4f113\") "
Nov 03 10:56:21.285668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.285640    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-openshift-ca" pod "c14091ff-a442-44ec-a0f5-00cc4ca4f113" (UID: "c14091ff-a442-44ec-a0f5-00cc4ca4f113"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:21.286132 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.286111    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:21.288106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.288073    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-xks-ca" pod "c14091ff-a442-44ec-a0f5-00cc4ca4f113" (UID: "c14091ff-a442-44ec-a0f5-00cc4ca4f113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:56:21.288673 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.288649    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14091ff-a442-44ec-a0f5-00cc4ca4f113-kube-api-access-grvgg" pod "c14091ff-a442-44ec-a0f5-00cc4ca4f113" (UID: "c14091ff-a442-44ec-a0f5-00cc4ca4f113"). InnerVolumeSpecName "kube-api-access-grvgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:21.388347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.388316    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14091ff-a442-44ec-a0f5-00cc4ca4f113-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:21.388347 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.388346    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvgg\" (UniqueName: \"kubernetes.io/projected/c14091ff-a442-44ec-a0f5-00cc4ca4f113-kube-api-access-grvgg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:21.429437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.429403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:21.530134 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.530105    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"]
Nov 03 10:56:21.530771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.530754    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c14091ff-a442-44ec-a0f5-00cc4ca4f113" containerName="revoke-keys"
Nov 03 10:56:21.530771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.530771    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14091ff-a442-44ec-a0f5-00cc4ca4f113" containerName="revoke-keys"
Nov 03 10:56:21.530956 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.530860    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14091ff-a442-44ec-a0f5-00cc4ca4f113" containerName="revoke-keys"
Nov 03 10:56:21.534527 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.534504    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.541807 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.541743    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"]
Nov 03 10:56:21.590755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.590724    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.590892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.590788    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.590957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.590910    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l4vh\" (UniqueName: \"kubernetes.io/projected/5aa9357a-6d83-496b-bb36-adfb70da00ec-kube-api-access-5l4vh\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.692727 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.692687    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.693214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.692780    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.693214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.692837    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l4vh\" (UniqueName: \"kubernetes.io/projected/5aa9357a-6d83-496b-bb36-adfb70da00ec-kube-api-access-5l4vh\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.693417 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.693384    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.695411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.695386    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.701125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.701098    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l4vh\" (UniqueName: \"kubernetes.io/projected/5aa9357a-6d83-496b-bb36-adfb70da00ec-kube-api-access-5l4vh\") pod \"maas-api-revoke-sub-d40a0af9-z8gnn\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.855291 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.855212    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:21.963616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.963487    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"
Nov 03 10:56:21.963616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.963514    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m" event={"ID":"c14091ff-a442-44ec-a0f5-00cc4ca4f113","Type":"ContainerDied","Data":"a5b148d65f4946072db2df69daa7690e7a8e84878c081d5887eec90b5fea0b07"}
Nov 03 10:56:21.963616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:21.963591    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5b148d65f4946072db2df69daa7690e7a8e84878c081d5887eec90b5fea0b07"
Nov 03 10:56:22.153046 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:56:22.153015    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa9357a_6d83_496b_bb36_adfb70da00ec.slice/crio-a9f1d8b8d337ea2d45d7f20451db7688fea8e6e9818ba75ae0229fce1b86f456 WatchSource:0}: Error finding container a9f1d8b8d337ea2d45d7f20451db7688fea8e6e9818ba75ae0229fce1b86f456: Status 404 returned error can't find the container with id a9f1d8b8d337ea2d45d7f20451db7688fea8e6e9818ba75ae0229fce1b86f456
Nov 03 10:56:22.153918 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:22.153897    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"]
Nov 03 10:56:22.377160 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:22.370596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:22.437709 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:56:22.437678    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:56:22.883857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:22.883824    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:56:22.973701 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:22.973660    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn" event={"ID":"5aa9357a-6d83-496b-bb36-adfb70da00ec","Type":"ContainerStarted","Data":"f6dbbe065832a171cc3b377bb67ed470b13fe10ec809a189a88ff2e46d583671"}
Nov 03 10:56:22.973858 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:22.973721    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn" event={"ID":"5aa9357a-6d83-496b-bb36-adfb70da00ec","Type":"ContainerStarted","Data":"a9f1d8b8d337ea2d45d7f20451db7688fea8e6e9818ba75ae0229fce1b86f456"}
Nov 03 10:56:22.990736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:22.990688    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn" podStartSLOduration=1.9906747889999998 podStartE2EDuration="1.990674789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:22.988374184 +0000 UTC m=+1613.193956236" watchObservedRunningTime="2026-11-03 10:56:22.990674789 +0000 UTC m=+1613.196256840"
Nov 03 10:56:22.995801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:22.995772    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:23.132600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:23.132105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:23.479249 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:23.478920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:24.162595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.160861    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:24.181649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.167324    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:56:24.280598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.275365    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"]
Nov 03 10:56:24.280598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.276225    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy"
Nov 03 10:56:24.280598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.276246    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy"
Nov 03 10:56:24.280598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.276433    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerName="istio-proxy"
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.291830    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-certs\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.291883    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-credential-socket\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.291978    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-token\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-token\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.292107    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-data\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.292168    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-podinfo\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.292213    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istiod-ca-cert\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.292261    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-envoy\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.292356    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-socket\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.294606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.292389    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-kube-api-access-8x7cp\") pod \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\" (UID: \"e2b1448c-b3f3-4409-8cf0-0ef57081b431\") "
Nov 03 10:56:24.373940 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.344999    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-credential-socket" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:24.401598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.374409    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istiod-ca-cert" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:24.401598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.382698    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-certs" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:24.401598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.367152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:24.429593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.411709    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-socket" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:24.464106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.462610    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-token" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:24.464106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.463932    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.464106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.463959    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.464106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.463975    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.464106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.464002    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.464106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.464016    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.590601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.547088    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-data" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:24.590601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.549999    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:24.590601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.576412    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.610612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.609875    5817 generic.go:356] "Generic (PLEG): container finished" podID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" containerID="8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a" exitCode=0
Nov 03 10:56:24.610612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.610190    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"
Nov 03 10:56:24.673106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.639968    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-envoy" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:24.673106 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.640091    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-podinfo" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:56:24.676496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.674687    5817 generic.go:356] "Generic (PLEG): container finished" podID="5aa9357a-6d83-496b-bb36-adfb70da00ec" containerID="f6dbbe065832a171cc3b377bb67ed470b13fe10ec809a189a88ff2e46d583671" exitCode=0
Nov 03 10:56:24.676496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.675421    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:56:24.683596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.678251    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.683596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.678284    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b1448c-b3f3-4409-8cf0-0ef57081b431-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.780749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.780658    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" event={"ID":"e2b1448c-b3f3-4409-8cf0-0ef57081b431","Type":"ContainerDied","Data":"8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a"}
Nov 03 10:56:24.781010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.780986    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"]
Nov 03 10:56:24.781167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.781147    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg" event={"ID":"e2b1448c-b3f3-4409-8cf0-0ef57081b431","Type":"ContainerDied","Data":"44f6bd70bdf054f9f838e524eb846a579dd79c56982ab7fface192bec0422acc"}
Nov 03 10:56:24.781308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.781290    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn" event={"ID":"5aa9357a-6d83-496b-bb36-adfb70da00ec","Type":"ContainerDied","Data":"f6dbbe065832a171cc3b377bb67ed470b13fe10ec809a189a88ff2e46d583671"}
Nov 03 10:56:24.781430 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.781417    5817 scope.go:122] "RemoveContainer" containerID="8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a"
Nov 03 10:56:24.781870 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.781849    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:24.817382 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.810437    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-kube-api-access-8x7cp" pod "e2b1448c-b3f3-4409-8cf0-0ef57081b431" (UID: "e2b1448c-b3f3-4409-8cf0-0ef57081b431"). InnerVolumeSpecName "kube-api-access-8x7cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:24.837809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.835497    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-fcc2ce-openshift-default-dockercfg-qkhp9\""
Nov 03 10:56:24.837809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.836353    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:24.897718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.885081    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/e2b1448c-b3f3-4409-8cf0-0ef57081b431-kube-api-access-8x7cp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:24.982841 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.970280    5817 scope.go:122] "RemoveContainer" containerID="8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a"
Nov 03 10:56:25.003944 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:56:24.990028    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a\": container with ID starting with 8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a not found: ID does not exist" containerID="8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a"
Nov 03 10:56:25.003944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:24.990076    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a"} err="failed to get container status \"8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a\": rpc error: code = NotFound desc = could not find container \"8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a\": container with ID starting with 8526e3b28ef32ace2a504ca01fde1dfceaf9d41fdfec2df348413205fc965b9a not found: ID does not exist"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.044129    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-data\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.046917    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlwf\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-kube-api-access-bzlwf\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.047018    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.047072    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-certs\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.047236    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-credential-socket\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.047290    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-token\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.047327    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-socket\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.047403    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-envoy\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.097933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.047479    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-podinfo\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.122700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.122658    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:25.167736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.155904    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-envoy\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.168806 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.168780    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-podinfo\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.169464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.169445    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-data\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.170327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.169997    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlwf\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-kube-api-access-bzlwf\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.170546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.169927    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-data\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.170783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.170762    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.171623 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.171532    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.171798 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.171778    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-certs\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.172270 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.172146    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-certs\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.172477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.172459    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-credential-socket\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.172956 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.172837    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-credential-socket\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.173338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.173318    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-token\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.173494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.173478    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-socket\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.188258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.187204    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-socket\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.192179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.191659    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:25.204452 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.204427    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-envoy\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.209604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.207061    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-podinfo\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.209604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.207469    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-token\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.243603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.240305    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlwf\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-kube-api-access-bzlwf\") pod \"e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") " pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.296152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.296079    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"]
Nov 03 10:56:25.324825 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.324791    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zw9xg"]
Nov 03 10:56:25.368409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.368108    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:25.462928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.462878    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:25.680666 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.679496    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:25.714651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.703160    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:56:25.856597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.840834    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:25.858184 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.858155    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:56:25.924665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.918450    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:25.984690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:25.982686    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 10:56:26.006776 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.006728    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.167826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.144304    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.235043 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.222938    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:56:26.259370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.259243    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Nov 03 10:56:26.501842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.500373    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 10:56:26.567853 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:56:26.567710    5817 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b1448c_b3f3_4409_8cf0_0ef57081b431.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 10:56:26.590182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.590147    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b1448c-b3f3-4409-8cf0-0ef57081b431" path="/var/lib/kubelet/pods/e2b1448c-b3f3-4409-8cf0-0ef57081b431/volumes"
Nov 03 10:56:26.590812 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.590789    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 10:56:26.590938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.590893    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:56:26.591012 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.590952    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:56:26.930792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.930750    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.932424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.932399    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:56:26.933909 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.933882    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.935682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.935657    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.958816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.954199    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 03 10:56:26.982856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.982732    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:26.983656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.983621    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.985253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.985231    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:56:26.986275 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.986226    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.995592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:26.986711    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:27.063751 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.063711    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"]
Nov 03 10:56:27.082697 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.082651    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:27.082968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.082933    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:27.098342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.097800    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:27.158090 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.158019    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:56:27.178742 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.163096    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 10:56:27.178742 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.167945    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:56:27.181268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.181245    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:56:27.183189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.183173    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:56:27.308593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.285853    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 10:56:27.335725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.335680    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" event={"ID":"7cb2e513-e727-4117-90c0-0c33f7feb2c6","Type":"ContainerStarted","Data":"ccd5db0625a61f52ff42dafc6e283a295122ce8e8193cddafc3f3b25ec61b5b1"}
Nov 03 10:56:27.574549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.574524    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:27.632139 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.632106    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-openshift-ca\") pod \"5aa9357a-6d83-496b-bb36-adfb70da00ec\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") "
Nov 03 10:56:27.632462 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.632411    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa9357a-6d83-496b-bb36-adfb70da00ec-kube-api-access-5l4vh\" (UniqueName: \"kubernetes.io/projected/5aa9357a-6d83-496b-bb36-adfb70da00ec-kube-api-access-5l4vh\") pod \"5aa9357a-6d83-496b-bb36-adfb70da00ec\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") "
Nov 03 10:56:27.632863 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.632833    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-xks-ca\") pod \"5aa9357a-6d83-496b-bb36-adfb70da00ec\" (UID: \"5aa9357a-6d83-496b-bb36-adfb70da00ec\") "
Nov 03 10:56:27.641303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.632473    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-openshift-ca" pod "5aa9357a-6d83-496b-bb36-adfb70da00ec" (UID: "5aa9357a-6d83-496b-bb36-adfb70da00ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:27.641437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.639864    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-xks-ca" pod "5aa9357a-6d83-496b-bb36-adfb70da00ec" (UID: "5aa9357a-6d83-496b-bb36-adfb70da00ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:56:27.641538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.641479    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa9357a-6d83-496b-bb36-adfb70da00ec-kube-api-access-5l4vh" pod "5aa9357a-6d83-496b-bb36-adfb70da00ec" (UID: "5aa9357a-6d83-496b-bb36-adfb70da00ec"). InnerVolumeSpecName "kube-api-access-5l4vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:27.743598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.742813    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:27.743598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.742852    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l4vh\" (UniqueName: \"kubernetes.io/projected/5aa9357a-6d83-496b-bb36-adfb70da00ec-kube-api-access-5l4vh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:27.743598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.742872    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa9357a-6d83-496b-bb36-adfb70da00ec-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:27.752586 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:27.750501    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:28.327596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.326904    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"
Nov 03 10:56:28.327596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.326938    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn" event={"ID":"5aa9357a-6d83-496b-bb36-adfb70da00ec","Type":"ContainerDied","Data":"a9f1d8b8d337ea2d45d7f20451db7688fea8e6e9818ba75ae0229fce1b86f456"}
Nov 03 10:56:28.327596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.326996    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f1d8b8d337ea2d45d7f20451db7688fea8e6e9818ba75ae0229fce1b86f456"
Nov 03 10:56:28.329633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.329604    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" event={"ID":"7cb2e513-e727-4117-90c0-0c33f7feb2c6","Type":"ContainerStarted","Data":"4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d"}
Nov 03 10:56:28.355719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.355677    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" podStartSLOduration=5.355664958 podStartE2EDuration="5.355664958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:28.350290737 +0000 UTC m=+1618.555872791" watchObservedRunningTime="2026-11-03 10:56:28.355664958 +0000 UTC m=+1618.561247008"
Nov 03 10:56:28.369535 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.369513    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:28.374552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.374524    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:28.563976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.563539    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:28.947803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:28.947763    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:29.336592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.336497    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"]
Nov 03 10:56:29.337524 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.337501    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5aa9357a-6d83-496b-bb36-adfb70da00ec" containerName="revoke-keys"
Nov 03 10:56:29.337651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.337526    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa9357a-6d83-496b-bb36-adfb70da00ec" containerName="revoke-keys"
Nov 03 10:56:29.337719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.337669    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa9357a-6d83-496b-bb36-adfb70da00ec" containerName="revoke-keys"
Nov 03 10:56:29.342791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.342498    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"]
Nov 03 10:56:29.342791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.342649    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.349145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.348241    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-47a8a4-openshift-default-dockercfg-97wwj\""
Nov 03 10:56:29.352232 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.351775    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:29.353541 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.353520    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:56:29.367105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367078    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.367225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367122    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.367295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367251    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.367295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367289    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.367432 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367329    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.367553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367529    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.367648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367627    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.367713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367678    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8422c\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-kube-api-access-8422c\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.367770 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.367734    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.369281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.369259    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.476935    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.477039    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.477076    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.477193    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.477229    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.477270    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.477355    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.477409    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.485596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.477456    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8422c\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-kube-api-access-8422c\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.653530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.577157    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.653530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.577409    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.653530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.577661    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.653530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.600854    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.653530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.618297    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.653530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.618591    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:29.658760 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.658675    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.659323 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.659138    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.660120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.660061    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.664713 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.663224    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8422c\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-kube-api-access-8422c\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:29.759900 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.759862    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:29.962460 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.962422    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-47a8a4-openshift-default-dockercfg-97wwj\""
Nov 03 10:56:29.970552 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:29.970523    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:30.319684 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:30.319606    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"]
Nov 03 10:56:30.322112 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:56:30.321099    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c730b7_3e59_4d9b_a03f_ff1fab06aaed.slice/crio-5cc609b5740ff38605cddccce951d67f211962949ec63268de0e381de6032dbc WatchSource:0}: Error finding container 5cc609b5740ff38605cddccce951d67f211962949ec63268de0e381de6032dbc: Status 404 returned error can't find the container with id 5cc609b5740ff38605cddccce951d67f211962949ec63268de0e381de6032dbc
Nov 03 10:56:30.324460 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:30.324424    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:30.324539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:30.324510    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:30.324611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:30.324551    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:30.362284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:30.362256    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" event={"ID":"76c730b7-3e59-4d9b-a03f-ff1fab06aaed","Type":"ContainerStarted","Data":"5cc609b5740ff38605cddccce951d67f211962949ec63268de0e381de6032dbc"}
Nov 03 10:56:30.463835 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:30.463798    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:30.919200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:30.919165    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:31.080780 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.080736    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:31.269674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.264188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:31.372702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.372658    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" event={"ID":"76c730b7-3e59-4d9b-a03f-ff1fab06aaed","Type":"ContainerStarted","Data":"bcc018edb94040e6656fe04c887ee40e4e495f3716cfe0104fa67b13dbab3008"}
Nov 03 10:56:31.416850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.416797    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" podStartSLOduration=2.416781347 podStartE2EDuration="2.416781347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:31.412227364 +0000 UTC m=+1621.617809415" watchObservedRunningTime="2026-11-03 10:56:31.416781347 +0000 UTC m=+1621.622363397"
Nov 03 10:56:31.557657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.554552    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:31.568021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.567990    5817 scope.go:122] "RemoveContainer" containerID="68ab68ba2f9d3118ab75bae0611d2245fe0499ba2224ac58ea51b26d8ad98f70"
Nov 03 10:56:31.713593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.712711    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:31.860180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.860041    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:31.970873 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.970836    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:31.978797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:31.978767    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:32.217634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.204099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:32.220141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.220117    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"]
Nov 03 10:56:32.226367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.226339    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:32.231085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.231057    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-fcc2ce-dockercfg-bmkhh\""
Nov 03 10:56:32.231212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.231146    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-fcc2ce\""
Nov 03 10:56:32.233467 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.233442    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"]
Nov 03 10:56:32.279501 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.279457    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33600aaa-9f93-471c-951b-54362ed05f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw\" (UID: \"33600aaa-9f93-471c-951b-54362ed05f01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:32.279703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.279523    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlg8\" (UniqueName: \"kubernetes.io/projected/33600aaa-9f93-471c-951b-54362ed05f01-kube-api-access-xnlg8\") pod \"payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw\" (UID: \"33600aaa-9f93-471c-951b-54362ed05f01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:32.355597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.344444    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:32.393100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.381875    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33600aaa-9f93-471c-951b-54362ed05f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw\" (UID: \"33600aaa-9f93-471c-951b-54362ed05f01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:32.393100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.381943    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlg8\" (UniqueName: \"kubernetes.io/projected/33600aaa-9f93-471c-951b-54362ed05f01-kube-api-access-xnlg8\") pod \"payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw\" (UID: \"33600aaa-9f93-471c-951b-54362ed05f01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:32.393100 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.383316    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33600aaa-9f93-471c-951b-54362ed05f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw\" (UID: \"33600aaa-9f93-471c-951b-54362ed05f01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:32.406425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.395790    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:32.406425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.397479    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:56:32.417593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.414304    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlg8\" (UniqueName: \"kubernetes.io/projected/33600aaa-9f93-471c-951b-54362ed05f01-kube-api-access-xnlg8\") pod \"payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw\" (UID: \"33600aaa-9f93-471c-951b-54362ed05f01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:32.501133 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.483953    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:32.561596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.559862    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:32.614608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.613776    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:56:32.614608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.613874    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:32.667720 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.663618    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"]
Nov 03 10:56:32.704262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.704233    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:32.720839 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.720801    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"]
Nov 03 10:56:32.720990 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.720923    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.780083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.779782    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:56:32.780083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.779928    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:56:32.803801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.803756    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.803970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.803816    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.803970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.803882    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nccc\" (UniqueName: \"kubernetes.io/projected/cbf59a21-0e48-4c45-abea-047c9efc3a6e-kube-api-access-4nccc\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.895866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.895835    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"]
Nov 03 10:56:32.906142 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.906119    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.906268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.906155    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.906268 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.906208    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nccc\" (UniqueName: \"kubernetes.io/projected/cbf59a21-0e48-4c45-abea-047c9efc3a6e-kube-api-access-4nccc\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.907304 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.907280    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.910496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.910450    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:32.914982 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:32.914945    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nccc\" (UniqueName: \"kubernetes.io/projected/cbf59a21-0e48-4c45-abea-047c9efc3a6e-kube-api-access-4nccc\") pod \"maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:33.053972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.053879    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:33.206771 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.204635    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:33.351430 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:56:33.351395    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf59a21_0e48_4c45_abea_047c9efc3a6e.slice/crio-ac65142f5778217eba37227e917d3f96873f2c5fa992763c29daf31b8ba14c45 WatchSource:0}: Error finding container ac65142f5778217eba37227e917d3f96873f2c5fa992763c29daf31b8ba14c45: Status 404 returned error can't find the container with id ac65142f5778217eba37227e917d3f96873f2c5fa992763c29daf31b8ba14c45
Nov 03 10:56:33.352251 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.352229    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"]
Nov 03 10:56:33.408913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.407620    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw" event={"ID":"33600aaa-9f93-471c-951b-54362ed05f01","Type":"ContainerStarted","Data":"93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317"}
Nov 03 10:56:33.408913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.407683    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw" event={"ID":"33600aaa-9f93-471c-951b-54362ed05f01","Type":"ContainerStarted","Data":"02a7324464a9d4f96848dcd7381ecf531e7f3f966ba5d32c126f45720b00eada"}
Nov 03 10:56:33.408913 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.407735    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:33.410534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.410510    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm" event={"ID":"cbf59a21-0e48-4c45-abea-047c9efc3a6e","Type":"ContainerStarted","Data":"ac65142f5778217eba37227e917d3f96873f2c5fa992763c29daf31b8ba14c45"}
Nov 03 10:56:33.432253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.432204    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw" podStartSLOduration=1.4321863559999999 podStartE2EDuration="1.432186356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:33.425257976 +0000 UTC m=+1623.630840025" watchObservedRunningTime="2026-11-03 10:56:33.432186356 +0000 UTC m=+1623.637768407"
Nov 03 10:56:33.566222 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.566191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:33.723680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.723644    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:33.857903 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:33.857822    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:34.424097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:34.424068    5817 generic.go:356] "Generic (PLEG): container finished" podID="cbf59a21-0e48-4c45-abea-047c9efc3a6e" containerID="e08b0a2dbbc97fd38c54a2880d03c2c0b15ea3719e26f062c209a79b98ee1c39" exitCode=0
Nov 03 10:56:34.424530 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:34.424164    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm" event={"ID":"cbf59a21-0e48-4c45-abea-047c9efc3a6e","Type":"ContainerDied","Data":"e08b0a2dbbc97fd38c54a2880d03c2c0b15ea3719e26f062c209a79b98ee1c39"}
Nov 03 10:56:34.624924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:34.622791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:35.390320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.390200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:35.437941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.437907    5817 generic.go:356] "Generic (PLEG): container finished" podID="061e1506-6c9f-4227-8f26-567737295e48" containerID="a75baeef46cac0b4aa16d284aa3713cae682bcfef6a615a3ba6d55c6b89a7743" exitCode=0
Nov 03 10:56:35.438448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.438210    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" event={"ID":"061e1506-6c9f-4227-8f26-567737295e48","Type":"ContainerDied","Data":"a75baeef46cac0b4aa16d284aa3713cae682bcfef6a615a3ba6d55c6b89a7743"}
Nov 03 10:56:35.439080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.438738    5817 scope.go:122] "RemoveContainer" containerID="a75baeef46cac0b4aa16d284aa3713cae682bcfef6a615a3ba6d55c6b89a7743"
Nov 03 10:56:35.440832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.440795    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:35.440941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.440868    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:35.440941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.440906    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:35.648649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.647008    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:35.791350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.791319    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf59a21-0e48-4c45-abea-047c9efc3a6e-kube-api-access-4nccc\" (UniqueName: \"kubernetes.io/projected/cbf59a21-0e48-4c45-abea-047c9efc3a6e-kube-api-access-4nccc\") pod \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") "
Nov 03 10:56:35.791557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.791407    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-openshift-ca\") pod \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") "
Nov 03 10:56:35.791649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.791615    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-xks-ca\") pod \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\" (UID: \"cbf59a21-0e48-4c45-abea-047c9efc3a6e\") "
Nov 03 10:56:35.791791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.791761    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-openshift-ca" pod "cbf59a21-0e48-4c45-abea-047c9efc3a6e" (UID: "cbf59a21-0e48-4c45-abea-047c9efc3a6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:35.792281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.792254    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:35.794072 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.794044    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-xks-ca" pod "cbf59a21-0e48-4c45-abea-047c9efc3a6e" (UID: "cbf59a21-0e48-4c45-abea-047c9efc3a6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:56:35.797431 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.797398    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf59a21-0e48-4c45-abea-047c9efc3a6e-kube-api-access-4nccc" pod "cbf59a21-0e48-4c45-abea-047c9efc3a6e" (UID: "cbf59a21-0e48-4c45-abea-047c9efc3a6e"). InnerVolumeSpecName "kube-api-access-4nccc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:35.894068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.894032    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf59a21-0e48-4c45-abea-047c9efc3a6e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:35.894068 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:35.894065    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nccc\" (UniqueName: \"kubernetes.io/projected/cbf59a21-0e48-4c45-abea-047c9efc3a6e-kube-api-access-4nccc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:36.220110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:36.220075    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:36.448522 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:36.448485    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" event={"ID":"061e1506-6c9f-4227-8f26-567737295e48","Type":"ContainerStarted","Data":"8f4aaa1dfa226e832654b437e98f476f8265bff606701159618bf3982058b364"}
Nov 03 10:56:36.450141 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:36.450116    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm" event={"ID":"cbf59a21-0e48-4c45-abea-047c9efc3a6e","Type":"ContainerDied","Data":"ac65142f5778217eba37227e917d3f96873f2c5fa992763c29daf31b8ba14c45"}
Nov 03 10:56:36.450342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:36.450161    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac65142f5778217eba37227e917d3f96873f2c5fa992763c29daf31b8ba14c45"
Nov 03 10:56:36.450342 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:36.450126    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"
Nov 03 10:56:36.970896 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:36.970863    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:37.038663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.038627    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"]
Nov 03 10:56:37.043508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.043484    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c8b50e-kmhpp"]
Nov 03 10:56:37.177608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.175898    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:37.180484 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.180458    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:56:37.180870 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.180844    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 10:56:37.180969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.180898    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 10:56:37.227786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.227712    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"]
Nov 03 10:56:37.228883 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.228862    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cbf59a21-0e48-4c45-abea-047c9efc3a6e" containerName="revoke-keys"
Nov 03 10:56:37.228883 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.228883    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf59a21-0e48-4c45-abea-047c9efc3a6e" containerName="revoke-keys"
Nov 03 10:56:37.229033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.228991    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf59a21-0e48-4c45-abea-047c9efc3a6e" containerName="revoke-keys"
Nov 03 10:56:37.234983 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.234959    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.238923 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.237559    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"]
Nov 03 10:56:37.239534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.239512    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:56:37.240271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.240251    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:56:37.328960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.325169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:37.339622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.339563    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.339763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.339665    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.339763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.339717    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh82d\" (UniqueName: \"kubernetes.io/projected/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-kube-api-access-gh82d\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.441244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.441215    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.441399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.441284    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh82d\" (UniqueName: \"kubernetes.io/projected/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-kube-api-access-gh82d\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.441399 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.441398    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.442233 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.442211    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.444033 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.444012    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.452093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.452070    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh82d\" (UniqueName: \"kubernetes.io/projected/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-kube-api-access-gh82d\") pod \"maas-api-revoke-sub-5cd9a19e-q9wq6\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.563631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.563503    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:37.882152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.882071    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:37.961678 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.961653    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2268cdc0-4a5a-467e-858e-9924436e84b0" path="/var/lib/kubelet/pods/2268cdc0-4a5a-467e-858e-9924436e84b0/volumes"
Nov 03 10:56:37.962049 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:37.962031    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"]
Nov 03 10:56:38.180969 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.180939    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 10:56:38.181093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.181001    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 10:56:38.353553 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.353527    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"]
Nov 03 10:56:38.359994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.359964    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:38.365694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.365669    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-cc8214\""
Nov 03 10:56:38.368899 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.368872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:38.373140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.373118    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"]
Nov 03 10:56:38.512612 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.503802    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqft6\" (UniqueName: \"kubernetes.io/projected/e5f470b0-c9e6-442d-922f-28562581b49f-kube-api-access-bqft6\") pod \"payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v\" (UID: \"e5f470b0-c9e6-442d-922f-28562581b49f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:38.532595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.518734    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f470b0-c9e6-442d-922f-28562581b49f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v\" (UID: \"e5f470b0-c9e6-442d-922f-28562581b49f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:38.588732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.586872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:38.590004 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.589978    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6" event={"ID":"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1","Type":"ContainerStarted","Data":"1bdfd8232f9ff680032a4748aa977a956eb58b570b2e284062e2c89117b6a8f3"}
Nov 03 10:56:38.590197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.590182    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6" event={"ID":"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1","Type":"ContainerStarted","Data":"2d50b7ba0790deabbca4bf5a99c88d70970418967ce1724ddd0432050cd6ac6e"}
Nov 03 10:56:38.623439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.623379    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6" podStartSLOduration=1.6233594550000001 podStartE2EDuration="1.623359455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:38.621988961 +0000 UTC m=+1628.827571013" watchObservedRunningTime="2026-11-03 10:56:38.623359455 +0000 UTC m=+1628.828941511"
Nov 03 10:56:38.647288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.647249    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqft6\" (UniqueName: \"kubernetes.io/projected/e5f470b0-c9e6-442d-922f-28562581b49f-kube-api-access-bqft6\") pod \"payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v\" (UID: \"e5f470b0-c9e6-442d-922f-28562581b49f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:38.664110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.664077    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f470b0-c9e6-442d-922f-28562581b49f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v\" (UID: \"e5f470b0-c9e6-442d-922f-28562581b49f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:38.665565 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.665147    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f470b0-c9e6-442d-922f-28562581b49f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v\" (UID: \"e5f470b0-c9e6-442d-922f-28562581b49f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:38.684353 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.684329    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqft6\" (UniqueName: \"kubernetes.io/projected/e5f470b0-c9e6-442d-922f-28562581b49f-kube-api-access-bqft6\") pod \"payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v\" (UID: \"e5f470b0-c9e6-442d-922f-28562581b49f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:38.844715 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.844679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:38.982395 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.980749    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:39.002115 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:38.984034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:39.185597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.185487    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 10:56:39.185763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.185552    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 10:56:39.324427 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.324397    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"]
Nov 03 10:56:39.325906 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:56:39.325878    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f470b0_c9e6_442d_922f_28562581b49f.slice/crio-e644cd46c91bea4d92fcae2bc88669eb746025f23d40d0c0de1a5faed30886d6 WatchSource:0}: Error finding container e644cd46c91bea4d92fcae2bc88669eb746025f23d40d0c0de1a5faed30886d6: Status 404 returned error can't find the container with id e644cd46c91bea4d92fcae2bc88669eb746025f23d40d0c0de1a5faed30886d6
Nov 03 10:56:39.628524 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.628442    5817 generic.go:356] "Generic (PLEG): container finished" podID="26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1" containerID="1bdfd8232f9ff680032a4748aa977a956eb58b570b2e284062e2c89117b6a8f3" exitCode=0
Nov 03 10:56:39.628994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.628520    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6" event={"ID":"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1","Type":"ContainerDied","Data":"1bdfd8232f9ff680032a4748aa977a956eb58b570b2e284062e2c89117b6a8f3"}
Nov 03 10:56:39.630809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.630790    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v" event={"ID":"e5f470b0-c9e6-442d-922f-28562581b49f","Type":"ContainerStarted","Data":"e601c729431cb74112ecc4147072e6cee0cefa3050c54bb0ed4dccc7820048a5"}
Nov 03 10:56:39.630911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.630824    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v" event={"ID":"e5f470b0-c9e6-442d-922f-28562581b49f","Type":"ContainerStarted","Data":"e644cd46c91bea4d92fcae2bc88669eb746025f23d40d0c0de1a5faed30886d6"}
Nov 03 10:56:39.630911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.630859    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:39.670828 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.670782    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v" podStartSLOduration=1.670769577 podStartE2EDuration="1.670769577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:39.666316564 +0000 UTC m=+1629.871898616" watchObservedRunningTime="2026-11-03 10:56:39.670769577 +0000 UTC m=+1629.876351627"
Nov 03 10:56:39.759791 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:39.759757    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:40.181357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:40.181320    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 10:56:40.181544 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:40.181374    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 10:56:40.757057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:40.754852    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:41.186757 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.186719    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 10:56:41.187879 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.187839    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 10:56:41.261827 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.261789    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:41.364312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.364278    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-xks-ca\") pod \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") "
Nov 03 10:56:41.365120 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.365094    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-openshift-ca\") pod \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") "
Nov 03 10:56:41.365753 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.365636    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-openshift-ca" pod "26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1" (UID: "26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:41.365922 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.365903    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-kube-api-access-gh82d\" (UniqueName: \"kubernetes.io/projected/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-kube-api-access-gh82d\") pod \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\" (UID: \"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1\") "
Nov 03 10:56:41.387752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.387666    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:41.473802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.471631    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-kube-api-access-gh82d" pod "26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1" (UID: "26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1"). InnerVolumeSpecName "kube-api-access-gh82d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:41.473802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.471763    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-xks-ca" pod "26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1" (UID: "26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:56:41.490783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.490712    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:41.490783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.490749    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh82d\" (UniqueName: \"kubernetes.io/projected/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1-kube-api-access-gh82d\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:41.898830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.889765    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6" event={"ID":"26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1","Type":"ContainerDied","Data":"2d50b7ba0790deabbca4bf5a99c88d70970418967ce1724ddd0432050cd6ac6e"}
Nov 03 10:56:41.898830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.890001    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d50b7ba0790deabbca4bf5a99c88d70970418967ce1724ddd0432050cd6ac6e"
Nov 03 10:56:41.898830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.890037    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"
Nov 03 10:56:41.947627 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:41.946801    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:42.012596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.008661    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:56:42.042672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.042631    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:42.050605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.044723    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 10:56:42.172654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.162652    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:42.172654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.168639    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:42.201596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.190851    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:56:42.201596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.191913    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 10:56:42.201596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.191955    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 10:56:42.247597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.246101    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": context deadline exceeded"
Nov 03 10:56:42.255605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.254268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:42.304206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.303675    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:42.717595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.712412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:42.850426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.849804    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:42.967655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.966674    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:42.991596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.979325    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:56:42.991596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:42.991263    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 10:56:43.071680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.021762    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 10:56:43.071680 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.022297    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:56:43.099821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.087554    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:43.113719 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.090665    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.115635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.090701    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 10:56:43.116696 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.116677    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:56:43.116883 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.116870    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:56:43.164523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.163661    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.164891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.164868    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:56:43.167367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.165802    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.170436 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.168970    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.170436 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.169507    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:56:43.237645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.237505    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 10:56:43.330601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.316737    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:56:43.330601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.317350    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.330601 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.318518    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:56:43.352594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.341738    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 10:56:43.518659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.518538    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:56:43.565597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.560444    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:43.578596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.578402    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:56:43.692595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.681688    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 10:56:43.692595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.689547    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 10:56:43.723822 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.701997    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 10:56:43.798667 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.789304    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 10:56:43.834152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.818778    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:43.834152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.819465    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 10:56:43.852060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.835492    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 10:56:43.852060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.835547    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 10:56:43.852060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:43.835596    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 10:56:44.003522 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:44.003484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:44.431176 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:44.430555    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 10:56:44.435027 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:44.434992    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:45.120808 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:45.120771    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:45.264324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:45.262091    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:45.840538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:45.840496    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:47.176712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:47.176678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:47.316594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:47.316091    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:48.057284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.057244    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:48.197280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.197247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:48.375885 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.375813    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"]
Nov 03 10:56:48.376685 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.376662    5817 cpu_manager.go:414] "containerMap: removing container" podUID="26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1" containerName="revoke-keys"
Nov 03 10:56:48.376788 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.376687    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1" containerName="revoke-keys"
Nov 03 10:56:48.376842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.376830    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1" containerName="revoke-keys"
Nov 03 10:56:48.381109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.381082    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.387080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.387044    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"]
Nov 03 10:56:48.393952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.393934    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.396778 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.396759    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"]
Nov 03 10:56:48.403094 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.403076    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"]
Nov 03 10:56:48.444469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444429    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.444613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444552    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.444656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444621    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.444689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444673    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.444736 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444708    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.444777 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444732    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975edffc-3c93-4c22-a72b-161989a9b772-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.444777 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444769    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.444859 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444806    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.444905 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444857    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8s6k\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-kube-api-access-k8s6k\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.444941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444900    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.444979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444959    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.445019 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.444982    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975edffc-3c93-4c22-a72b-161989a9b772-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.445019 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.445008    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.445087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.445060    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.445126 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.445101    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xz4\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-kube-api-access-62xz4\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.445163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.445129    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.445199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.445161    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.445262 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.445246    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.546614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.546561    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.546795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.546648    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.546795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.546746    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.546960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.546796    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.546960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.546848    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.546960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.546912    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.547145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.546960    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975edffc-3c93-4c22-a72b-161989a9b772-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.547145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547000    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.547145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547054    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.547145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547127    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8s6k\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-kube-api-access-k8s6k\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547169    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547174    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547251    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547289    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975edffc-3c93-4c22-a72b-161989a9b772-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547541    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547654    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547695    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xz4\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-kube-api-access-62xz4\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547707    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547725    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.547725 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547720    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.551096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547774    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.551096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.547960    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.551096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.548027    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.551096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.549028    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975edffc-3c93-4c22-a72b-161989a9b772-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.551096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.549279    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.551096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.549756    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.551096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.550009    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.551096 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.550622    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.551712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.551679    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.552097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.552074    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.552298 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.552256    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.553151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.553130    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975edffc-3c93-4c22-a72b-161989a9b772-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.565846 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.565820    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.575634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.574666    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xz4\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-kube-api-access-62xz4\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.575634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.575592    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8s6k\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-kube-api-access-k8s6k\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:48.576241 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.576225    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.686089 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.686049    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:48.703475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.703444    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:48.779224 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:48.775877    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:49.084188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.084147    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.084271 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.084246    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.084306 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.084290    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.084648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.084628    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"]
Nov 03 10:56:49.119069 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.119024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:49.218971 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.218933    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" event={"ID":"257d4f7b-bf89-4f02-98f6-774fcd4994e3","Type":"ContainerStarted","Data":"5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925"}
Nov 03 10:56:49.219383 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.219004    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" event={"ID":"257d4f7b-bf89-4f02-98f6-774fcd4994e3","Type":"ContainerStarted","Data":"3694024a2e6796a87d0bcbadc8415a60c8222610e6ce0b0170c236c004f1d3ef"}
Nov 03 10:56:49.242174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.242060    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" podStartSLOduration=1.242044219 podStartE2EDuration="1.242044219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:49.241610834 +0000 UTC m=+1639.447192886" watchObservedRunningTime="2026-11-03 10:56:49.242044219 +0000 UTC m=+1639.447626273"
Nov 03 10:56:49.281943 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:56:49.281914    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975edffc_3c93_4c22_a72b_161989a9b772.slice/crio-a107fbf1952fb32fb71460cdc905a22fea20618c9340a229c70835bcc1e294d8 WatchSource:0}: Error finding container a107fbf1952fb32fb71460cdc905a22fea20618c9340a229c70835bcc1e294d8: Status 404 returned error can't find the container with id a107fbf1952fb32fb71460cdc905a22fea20618c9340a229c70835bcc1e294d8
Nov 03 10:56:49.282199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.282180    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"]
Nov 03 10:56:49.284538 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.284510    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.284639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.284587    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.284639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.284617    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.307203 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.307147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:49.707166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:49.707090    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:50.157675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.155086    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:50.269277 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.269243    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" event={"ID":"975edffc-3c93-4c22-a72b-161989a9b772","Type":"ContainerStarted","Data":"b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481"}
Nov 03 10:56:50.270004 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.269956    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" event={"ID":"975edffc-3c93-4c22-a72b-161989a9b772","Type":"ContainerStarted","Data":"a107fbf1952fb32fb71460cdc905a22fea20618c9340a229c70835bcc1e294d8"}
Nov 03 10:56:50.328967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.328899    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podStartSLOduration=2.328876625 podStartE2EDuration="2.328876625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:50.306297161 +0000 UTC m=+1640.511879212" watchObservedRunningTime="2026-11-03 10:56:50.328876625 +0000 UTC m=+1640.534458682"
Nov 03 10:56:50.657002 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.656971    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 10:56:50.763464 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.763425    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 10:56:50.763737 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.763707    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 10:56:50.783597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.780416    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:50.783597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.781534    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 10:56:50.783597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.781597    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 10:56:50.961655 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:50.949508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:51.714595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:51.713630    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 10:56:51.714595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:51.713689    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 10:56:51.775692 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:51.775657    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:51.782597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:51.781219    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 10:56:51.782597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:51.781314    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 10:56:52.461403 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:56:52.461374    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:56:52.717653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:52.713827    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 10:56:52.717653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:52.713884    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 10:56:52.777856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:52.777824    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 10:56:52.778013 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:52.777879    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 10:56:53.006649 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:53.000660    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:53.319652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:53.310353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:53.470386 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:53.470352    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:53.708602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:53.705521    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 10:56:53.708602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:53.705600    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 10:56:53.777836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:53.777801    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 10:56:53.778242 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:53.777856    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 10:56:54.081465 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.081383    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:54.394653 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.390685    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:54.717626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.710491    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 10:56:54.717626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.710556    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 10:56:54.732217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.732185    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:54.757411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.757380    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"]
Nov 03 10:56:54.796080 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.796043    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 10:56:54.796630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.796600    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 10:56:54.797042 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.796226    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a35a97-kt8wm"]
Nov 03 10:56:54.988657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:54.984832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:55.148722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.148689    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:55.183670 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.183644    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"]
Nov 03 10:56:55.253965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.253887    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:55.312364 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.311859    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ce5555\""
Nov 03 10:56:55.313402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.313112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:55.313402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.313301    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ce5555-dockercfg-tmnvs\""
Nov 03 10:56:55.560417 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.560189    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26pm\" (UniqueName: \"kubernetes.io/projected/da43d24c-95cb-4245-bf73-80269d37b022-kube-api-access-s26pm\") pod \"payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx\" (UID: \"da43d24c-95cb-4245-bf73-80269d37b022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:55.561368 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.561333    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d24c-95cb-4245-bf73-80269d37b022-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx\" (UID: \"da43d24c-95cb-4245-bf73-80269d37b022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:55.572694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.572660    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"]
Nov 03 10:56:55.577122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.577098    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"]
Nov 03 10:56:55.582517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.582495    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"]
Nov 03 10:56:55.582639 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.582625    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:55.665994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.665963    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7mxf\" (UniqueName: \"kubernetes.io/projected/0cc6fa9d-166b-489a-bf44-d9f1787d0146-kube-api-access-z7mxf\") pod \"payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442\" (UID: \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:55.666163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.666029    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26pm\" (UniqueName: \"kubernetes.io/projected/da43d24c-95cb-4245-bf73-80269d37b022-kube-api-access-s26pm\") pod \"payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx\" (UID: \"da43d24c-95cb-4245-bf73-80269d37b022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:55.666163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.666140    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d24c-95cb-4245-bf73-80269d37b022-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx\" (UID: \"da43d24c-95cb-4245-bf73-80269d37b022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:55.666341 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.666258    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc6fa9d-166b-489a-bf44-d9f1787d0146-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442\" (UID: \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:55.666844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.666825    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d24c-95cb-4245-bf73-80269d37b022-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx\" (UID: \"da43d24c-95cb-4245-bf73-80269d37b022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:55.673895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.673870    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26pm\" (UniqueName: \"kubernetes.io/projected/da43d24c-95cb-4245-bf73-80269d37b022-kube-api-access-s26pm\") pod \"payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx\" (UID: \"da43d24c-95cb-4245-bf73-80269d37b022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:55.721144 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.721114    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 10:56:55.721370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.721346    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 10:56:55.738439 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.738410    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:55.767339 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.767304    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc6fa9d-166b-489a-bf44-d9f1787d0146-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442\" (UID: \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:55.767592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.767553    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7mxf\" (UniqueName: \"kubernetes.io/projected/0cc6fa9d-166b-489a-bf44-d9f1787d0146-kube-api-access-z7mxf\") pod \"payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442\" (UID: \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:55.768997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.768972    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc6fa9d-166b-489a-bf44-d9f1787d0146-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442\" (UID: \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:55.776450 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.776431    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7mxf\" (UniqueName: \"kubernetes.io/projected/0cc6fa9d-166b-489a-bf44-d9f1787d0146-kube-api-access-z7mxf\") pod \"payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442\" (UID: \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:55.778171 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.778145    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 10:56:55.778257 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.778193    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 10:56:55.912930 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.911876    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf59a21-0e48-4c45-abea-047c9efc3a6e" path="/var/lib/kubelet/pods/cbf59a21-0e48-4c45-abea-047c9efc3a6e/volumes"
Nov 03 10:56:55.931911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:55.931885    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:56.065067 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.065033    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"]
Nov 03 10:56:56.068945 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:56:56.068877    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda43d24c_95cb_4245_bf73_80269d37b022.slice/crio-b7696950bbf7fbc293693f6b4c3ccd1ff2d1001af12e8475dfaec14b5c5d927e WatchSource:0}: Error finding container b7696950bbf7fbc293693f6b4c3ccd1ff2d1001af12e8475dfaec14b5c5d927e: Status 404 returned error can't find the container with id b7696950bbf7fbc293693f6b4c3ccd1ff2d1001af12e8475dfaec14b5c5d927e
Nov 03 10:56:56.155549 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.155518    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:56.246542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.246513    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"]
Nov 03 10:56:56.304248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.304152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:56.459065 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.459031    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:56.596372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.596237    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442" event={"ID":"0cc6fa9d-166b-489a-bf44-d9f1787d0146","Type":"ContainerStarted","Data":"8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d"}
Nov 03 10:56:56.596372 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.596304    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442" event={"ID":"0cc6fa9d-166b-489a-bf44-d9f1787d0146","Type":"ContainerStarted","Data":"676fcf78b323a87e572b2dacbb2053a5cf8d2e6e0d982fea11ff6c816a469236"}
Nov 03 10:56:56.596614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.596500    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:56:56.598539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.598512    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx" event={"ID":"da43d24c-95cb-4245-bf73-80269d37b022","Type":"ContainerStarted","Data":"f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516"}
Nov 03 10:56:56.598702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.598557    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx" event={"ID":"da43d24c-95cb-4245-bf73-80269d37b022","Type":"ContainerStarted","Data":"b7696950bbf7fbc293693f6b4c3ccd1ff2d1001af12e8475dfaec14b5c5d927e"}
Nov 03 10:56:56.598702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.598623    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:56:56.618650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.618561    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442" podStartSLOduration=1.6185497789999999 podStartE2EDuration="1.618549779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:56.613186965 +0000 UTC m=+1646.818769018" watchObservedRunningTime="2026-11-03 10:56:56.618549779 +0000 UTC m=+1646.824131858"
Nov 03 10:56:56.636767 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.636700    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx" podStartSLOduration=1.636684653 podStartE2EDuration="1.636684653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:56.631761926 +0000 UTC m=+1646.837343978" watchObservedRunningTime="2026-11-03 10:56:56.636684653 +0000 UTC m=+1646.842266705"
Nov 03 10:56:56.777416 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.777384    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 10:56:56.777613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:56.777447    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 10:56:57.368396 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:57.368332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:57.410445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:57.410417    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:57.410984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:57.410964    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:57.411881 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:57.411861    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 10:56:57.548708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:57.547629    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:57.784105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:57.784060    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:57.784738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:57.784716    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:57.785388 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:57.785365    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 10:56:58.373764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:58.373728    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:59.081099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:59.081064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:59.418537 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:59.418461    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:56:59.989595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:56:59.983509    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:00.842116 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:00.842083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:00.974559 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:00.974525    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:01.343960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:01.343922    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:02.324880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:02.324843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:02.460303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:02.460256    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:02.710408 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:02.710363    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:03.339698 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:03.339663    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:03.886119 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:03.886091    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:57:04.069246 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.069214    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"]
Nov 03 10:57:04.085595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.082171    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.087600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.086788    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:57:04.087600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.086910    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:57:04.118105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.113621    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"]
Nov 03 10:57:04.203105 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.203065    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.203511 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.203464    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.203833 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.203801    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bv9\" (UniqueName: \"kubernetes.io/projected/6c66bc84-4b3a-434c-9233-8042710123b9-kube-api-access-d8bv9\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.305540 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.305495    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.305779 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.305626    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bv9\" (UniqueName: \"kubernetes.io/projected/6c66bc84-4b3a-434c-9233-8042710123b9-kube-api-access-d8bv9\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.305861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.305781    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.306282 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.306259    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.308523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.308504    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.313867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.313849    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bv9\" (UniqueName: \"kubernetes.io/projected/6c66bc84-4b3a-434c-9233-8042710123b9-kube-api-access-d8bv9\") pod \"maas-api-revoke-sub-d4160bfd-7nqcv\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.448600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.448550    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:04.577220 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.577179    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:04.813647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:04.806530    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"]
Nov 03 10:57:04.838708 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:57:04.836818    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c66bc84_4b3a_434c_9233_8042710123b9.slice/crio-784efa2fe35a6504a8b0c08567d9c9d64210c44e26447c31d3478e552f268220 WatchSource:0}: Error finding container 784efa2fe35a6504a8b0c08567d9c9d64210c44e26447c31d3478e552f268220: Status 404 returned error can't find the container with id 784efa2fe35a6504a8b0c08567d9c9d64210c44e26447c31d3478e552f268220
Nov 03 10:57:05.873596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:05.868204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:06.121449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:06.116110    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv" event={"ID":"6c66bc84-4b3a-434c-9233-8042710123b9","Type":"ContainerStarted","Data":"88cc6413f8ffc9228bb89c9a57c0f0b146637ddf1d609ebe20dc7a13505fd4f4"}
Nov 03 10:57:06.121449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:06.116169    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv" event={"ID":"6c66bc84-4b3a-434c-9233-8042710123b9","Type":"ContainerStarted","Data":"784efa2fe35a6504a8b0c08567d9c9d64210c44e26447c31d3478e552f268220"}
Nov 03 10:57:06.121449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:06.117888    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:06.145912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:06.144621    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"]
Nov 03 10:57:06.157194 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:06.157134    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv" podStartSLOduration=2.157116333 podStartE2EDuration="2.157116333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:06.133973162 +0000 UTC m=+1656.339555215" watchObservedRunningTime="2026-11-03 10:57:06.157116333 +0000 UTC m=+1656.362698384"
Nov 03 10:57:06.159721 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:06.157425    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2731157-k7jm9"]
Nov 03 10:57:07.132002 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:07.131960    5817 generic.go:356] "Generic (PLEG): container finished" podID="6c66bc84-4b3a-434c-9233-8042710123b9" containerID="88cc6413f8ffc9228bb89c9a57c0f0b146637ddf1d609ebe20dc7a13505fd4f4" exitCode=0
Nov 03 10:57:07.132494 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:07.132022    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv" event={"ID":"6c66bc84-4b3a-434c-9233-8042710123b9","Type":"ContainerDied","Data":"88cc6413f8ffc9228bb89c9a57c0f0b146637ddf1d609ebe20dc7a13505fd4f4"}
Nov 03 10:57:07.618933 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:07.618882    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 10:57:07.619144 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:07.618976    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 10:57:07.865213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:07.865178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:07.936188 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:07.936153    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647dd6bf-02ac-40e1-a22d-1528f1968467" path="/var/lib/kubelet/pods/647dd6bf-02ac-40e1-a22d-1528f1968467/volumes"
Nov 03 10:57:08.368137 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.368114    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:08.435318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.435281    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c66bc84-4b3a-434c-9233-8042710123b9-kube-api-access-d8bv9\" (UniqueName: \"kubernetes.io/projected/6c66bc84-4b3a-434c-9233-8042710123b9-kube-api-access-d8bv9\") pod \"6c66bc84-4b3a-434c-9233-8042710123b9\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") "
Nov 03 10:57:08.435516 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.435333    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-openshift-ca\") pod \"6c66bc84-4b3a-434c-9233-8042710123b9\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") "
Nov 03 10:57:08.435862 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.435749    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-openshift-ca" pod "6c66bc84-4b3a-434c-9233-8042710123b9" (UID: "6c66bc84-4b3a-434c-9233-8042710123b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:57:08.436099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.436048    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-xks-ca\") pod \"6c66bc84-4b3a-434c-9233-8042710123b9\" (UID: \"6c66bc84-4b3a-434c-9233-8042710123b9\") "
Nov 03 10:57:08.437036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.437012    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:08.438365 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.438339    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c66bc84-4b3a-434c-9233-8042710123b9-kube-api-access-d8bv9" pod "6c66bc84-4b3a-434c-9233-8042710123b9" (UID: "6c66bc84-4b3a-434c-9233-8042710123b9"). InnerVolumeSpecName "kube-api-access-d8bv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:57:08.438605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.438555    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-xks-ca" pod "6c66bc84-4b3a-434c-9233-8042710123b9" (UID: "6c66bc84-4b3a-434c-9233-8042710123b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:57:08.538175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.538083    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c66bc84-4b3a-434c-9233-8042710123b9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:08.538175 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.538115    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bv9\" (UniqueName: \"kubernetes.io/projected/6c66bc84-4b3a-434c-9233-8042710123b9-kube-api-access-d8bv9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:08.749294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:08.748749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:09.153044 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:09.153019    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"
Nov 03 10:57:09.153223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:09.153062    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv" event={"ID":"6c66bc84-4b3a-434c-9233-8042710123b9","Type":"ContainerDied","Data":"784efa2fe35a6504a8b0c08567d9c9d64210c44e26447c31d3478e552f268220"}
Nov 03 10:57:09.153223 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:09.153113    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="784efa2fe35a6504a8b0c08567d9c9d64210c44e26447c31d3478e552f268220"
Nov 03 10:57:09.534595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:09.527463    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:10.504400 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:10.504359    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:11.365707 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:11.358885    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:12.079716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:12.079679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:12.754164 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:12.754074    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:13.433693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:13.433661    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:13.574545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:13.574465    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:14.259297 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:14.259265    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:14.394964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:14.394788    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:14.738426 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:14.738390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:14.882921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:14.881048    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": read tcp 10.133.0.2:50712->10.133.0.183:15021: read: connection reset by peer" start-of-body=
Nov 03 10:57:14.882921 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:14.881109    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": read tcp 10.133.0.2:50712->10.133.0.183:15021: read: connection reset by peer"
Nov 03 10:57:15.202632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.197054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:15.337850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.337815    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:15.438640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.438605    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" event={"ID":"7cb2e513-e727-4117-90c0-0c33f7feb2c6","Type":"ContainerDied","Data":"4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d"}
Nov 03 10:57:15.438803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.438611    5817 generic.go:356] "Generic (PLEG): container finished" podID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerID="4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d" exitCode=0
Nov 03 10:57:15.439146 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.439132    5817 scope.go:122] "RemoveContainer" containerID="4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d"
Nov 03 10:57:15.440893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.440868    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:15.440996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.440958    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:15.440996 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.440989    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:15.724480 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.724411    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:15.961801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:15.961743    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:16.068559 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.068418    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"]
Nov 03 10:57:16.069150 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.069125    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6c66bc84-4b3a-434c-9233-8042710123b9" containerName="revoke-keys"
Nov 03 10:57:16.069150 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.069148    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c66bc84-4b3a-434c-9233-8042710123b9" containerName="revoke-keys"
Nov 03 10:57:16.069360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.069345    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c66bc84-4b3a-434c-9233-8042710123b9" containerName="revoke-keys"
Nov 03 10:57:16.072989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.072970    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.079359 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.079333    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-3ba121-openshift-default-dockercfg-xtchh\""
Nov 03 10:57:16.082087 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.082060    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"]
Nov 03 10:57:16.129302 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.129270    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.259701    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbqw\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-kube-api-access-stbqw\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.259765    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851221cb-59f2-424f-8c42-40104dafa17b-istio-podinfo\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.259817    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-credential-socket\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.259853    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-envoy\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.259876    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-istio-token\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.259903    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-certs\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.259933    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851221cb-59f2-424f-8c42-40104dafa17b-istiod-ca-cert\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.259973    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-data\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.273596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.260152    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-socket\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.298605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.295795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.367591    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-socket\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.367814    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbqw\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-kube-api-access-stbqw\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.367856    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851221cb-59f2-424f-8c42-40104dafa17b-istio-podinfo\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.367903    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-credential-socket\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.367938    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-envoy\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.367961    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-istio-token\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.367997    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-certs\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.368027    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851221cb-59f2-424f-8c42-40104dafa17b-istiod-ca-cert\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.368068    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-data\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.368739    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-data\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.369352    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-socket\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.372646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.370507    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-credential-socket\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.386941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.385863    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-certs\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.386941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.386320    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851221cb-59f2-424f-8c42-40104dafa17b-istiod-ca-cert\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.415031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.415001    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-envoy\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.429591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.423066    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851221cb-59f2-424f-8c42-40104dafa17b-istio-podinfo\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.429591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.423527    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-istio-token\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.429591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.429200    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbqw\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-kube-api-access-stbqw\") pod \"e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") " pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.443023 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.442992    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:16.443644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.443628    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:16.481895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.480761    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" event={"ID":"7cb2e513-e727-4117-90c0-0c33f7feb2c6","Type":"ContainerStarted","Data":"006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978"}
Nov 03 10:57:16.925736 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:57:16.925685    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851221cb_59f2_424f_8c42_40104dafa17b.slice/crio-36ce6f12c9632d2ced690206cbddad1912e2c8b6f77991afa42a27ea4dd91100 WatchSource:0}: Error finding container 36ce6f12c9632d2ced690206cbddad1912e2c8b6f77991afa42a27ea4dd91100: Status 404 returned error can't find the container with id 36ce6f12c9632d2ced690206cbddad1912e2c8b6f77991afa42a27ea4dd91100
Nov 03 10:57:16.927708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.927675    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"]
Nov 03 10:57:16.929975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.929942    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:16.930112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.930030    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:16.930112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:16.930076    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:17.050327 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:17.050288    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:17.369545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:17.369510    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:57:17.369759 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:17.369730    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 10:57:17.369817 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:17.369782    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 10:57:17.547895 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:17.547856    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" event={"ID":"851221cb-59f2-424f-8c42-40104dafa17b","Type":"ContainerStarted","Data":"bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380"}
Nov 03 10:57:17.548305 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:17.547939    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" event={"ID":"851221cb-59f2-424f-8c42-40104dafa17b","Type":"ContainerStarted","Data":"36ce6f12c9632d2ced690206cbddad1912e2c8b6f77991afa42a27ea4dd91100"}
Nov 03 10:57:17.569625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:17.569540    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" podStartSLOduration=1.569523035 podStartE2EDuration="1.569523035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:17.565219825 +0000 UTC m=+1667.770801902" watchObservedRunningTime="2026-11-03 10:57:17.569523035 +0000 UTC m=+1667.775105087"
Nov 03 10:57:18.170599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.162007    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:18.369423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.369386    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 10:57:18.369613 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.369437    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 10:57:18.453600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.453545    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:18.469997 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.467058    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:18.472122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.472095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:18.559972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.559938    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:18.561101 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.561079    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:57:18.950867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:18.950832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:19.254636 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:19.254518    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:19.369349 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:19.369297    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 10:57:19.369545 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:19.369374    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 10:57:19.414596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:19.414427    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:19.560659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:19.560057    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:20.032866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.032832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:20.163616 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.163583    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:20.370000 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.369920    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 10:57:20.370000 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.369977    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 10:57:20.377118 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.377091    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:20.844935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.844901    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:20.860765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.860738    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"]
Nov 03 10:57:20.866150 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.866129    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:20.872826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.872734    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e11de4-dockercfg-q8pwd\""
Nov 03 10:57:20.872959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.872836    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e11de4\""
Nov 03 10:57:20.874276 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.874016    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"]
Nov 03 10:57:20.941599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.941376    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"]
Nov 03 10:57:20.952642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.950032    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76n5\" (UniqueName: \"kubernetes.io/projected/f52cc2fc-370d-4a93-a725-225d36d0ec5d-kube-api-access-p76n5\") pod \"payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs\" (UID: \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:20.952642 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.950125    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f52cc2fc-370d-4a93-a725-225d36d0ec5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs\" (UID: \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:20.956668 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.956616    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"]
Nov 03 10:57:20.957021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:20.957006    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:21.021600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.020687    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:21.055597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.054796    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76n5\" (UniqueName: \"kubernetes.io/projected/f52cc2fc-370d-4a93-a725-225d36d0ec5d-kube-api-access-p76n5\") pod \"payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs\" (UID: \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:21.055597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.054865    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7t2s\" (UniqueName: \"kubernetes.io/projected/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-kube-api-access-x7t2s\") pod \"payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn\" (UID: \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:21.055597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.054889    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f52cc2fc-370d-4a93-a725-225d36d0ec5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs\" (UID: \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:21.055597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.055041    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn\" (UID: \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:21.059593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.057735    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f52cc2fc-370d-4a93-a725-225d36d0ec5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs\" (UID: \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:21.080415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.080379    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76n5\" (UniqueName: \"kubernetes.io/projected/f52cc2fc-370d-4a93-a725-225d36d0ec5d-kube-api-access-p76n5\") pod \"payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs\" (UID: \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:21.173654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.165612    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7t2s\" (UniqueName: \"kubernetes.io/projected/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-kube-api-access-x7t2s\") pod \"payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn\" (UID: \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:21.173654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.165795    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn\" (UID: \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:21.173654 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.166801    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn\" (UID: \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:21.182597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.181551    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7t2s\" (UniqueName: \"kubernetes.io/projected/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-kube-api-access-x7t2s\") pod \"payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn\" (UID: \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:21.229764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.227833    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:21.274671 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.274636    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:21.371024 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.369513    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 10:57:21.371024 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.369640    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 10:57:21.482154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.479733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:21.656724 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.656644    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:21.665561 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.665531    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"]
Nov 03 10:57:21.675963 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.675932    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:21.678006 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.675814    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"]
Nov 03 10:57:21.683214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.683130    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3ba121-dockercfg-mw486\""
Nov 03 10:57:21.683214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.683133    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3ba121\""
Nov 03 10:57:21.819117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.805395    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw65v\" (UniqueName: \"kubernetes.io/projected/7e17c72e-2eec-4714-841f-2763a09d054b-kube-api-access-sw65v\") pod \"payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8\" (UID: \"7e17c72e-2eec-4714-841f-2763a09d054b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:21.819117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.805527    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e17c72e-2eec-4714-841f-2763a09d054b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8\" (UID: \"7e17c72e-2eec-4714-841f-2763a09d054b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:21.825661 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.824101    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:21.932593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.931387    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw65v\" (UniqueName: \"kubernetes.io/projected/7e17c72e-2eec-4714-841f-2763a09d054b-kube-api-access-sw65v\") pod \"payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8\" (UID: \"7e17c72e-2eec-4714-841f-2763a09d054b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:21.932593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.931550    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e17c72e-2eec-4714-841f-2763a09d054b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8\" (UID: \"7e17c72e-2eec-4714-841f-2763a09d054b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:21.959599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:21.951339    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e17c72e-2eec-4714-841f-2763a09d054b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8\" (UID: \"7e17c72e-2eec-4714-841f-2763a09d054b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:21.999615 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:57:21.993106    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52cc2fc_370d_4a93_a725_225d36d0ec5d.slice/crio-2ab9c04ca1f0c2bccdcf2ad5086f7f2dd23bc893534ac44bfd7725b2a813ad75 WatchSource:0}: Error finding container 2ab9c04ca1f0c2bccdcf2ad5086f7f2dd23bc893534ac44bfd7725b2a813ad75: Status 404 returned error can't find the container with id 2ab9c04ca1f0c2bccdcf2ad5086f7f2dd23bc893534ac44bfd7725b2a813ad75
Nov 03 10:57:22.051617 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.051525    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw65v\" (UniqueName: \"kubernetes.io/projected/7e17c72e-2eec-4714-841f-2763a09d054b-kube-api-access-sw65v\") pod \"payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8\" (UID: \"7e17c72e-2eec-4714-841f-2763a09d054b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:22.064409 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.064379    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"]
Nov 03 10:57:22.064598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.064436    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"]
Nov 03 10:57:22.323773 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.323735    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:22.376224 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.374098    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:57:22.376224 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.374978    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:57:22.376457 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.376274    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 10:57:22.442847 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:57:22.441132    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:57:22.611025 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:57:22.610993    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e17c72e_2eec_4714_841f_2763a09d054b.slice/crio-e1338d5bd9d258fb247cc99c262f56136024bafce2247931b805fb1325976e7c WatchSource:0}: Error finding container e1338d5bd9d258fb247cc99c262f56136024bafce2247931b805fb1325976e7c: Status 404 returned error can't find the container with id e1338d5bd9d258fb247cc99c262f56136024bafce2247931b805fb1325976e7c
Nov 03 10:57:22.611370 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.611350    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"]
Nov 03 10:57:22.694652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.694618    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs" event={"ID":"f52cc2fc-370d-4a93-a725-225d36d0ec5d","Type":"ContainerStarted","Data":"4369926e61ac697fe7c3cbb71df5829b721416a0bd4f01d192dc33e72ffd1b4f"}
Nov 03 10:57:22.694796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.694688    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs" event={"ID":"f52cc2fc-370d-4a93-a725-225d36d0ec5d","Type":"ContainerStarted","Data":"2ab9c04ca1f0c2bccdcf2ad5086f7f2dd23bc893534ac44bfd7725b2a813ad75"}
Nov 03 10:57:22.694796 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.694737    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:22.697140 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.697111    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8" event={"ID":"7e17c72e-2eec-4714-841f-2763a09d054b","Type":"ContainerStarted","Data":"1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457"}
Nov 03 10:57:22.697258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.697164    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8" event={"ID":"7e17c72e-2eec-4714-841f-2763a09d054b","Type":"ContainerStarted","Data":"e1338d5bd9d258fb247cc99c262f56136024bafce2247931b805fb1325976e7c"}
Nov 03 10:57:22.697258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.697191    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:22.699555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.699532    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn" event={"ID":"dda1d70a-1612-4004-93bd-7ce1a4f3a28e","Type":"ContainerStarted","Data":"06c0d32b0e7d8b3b941d2ab37ee2006686877405b47a99316b561d1bfb321120"}
Nov 03 10:57:22.699682 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.699629    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn" event={"ID":"dda1d70a-1612-4004-93bd-7ce1a4f3a28e","Type":"ContainerStarted","Data":"cc0eb0684759083f6773863c76105ddd5a2ae396ef2fe52890bb512236fc354b"}
Nov 03 10:57:22.699792 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.699771    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:22.710973 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.710925    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs" podStartSLOduration=2.710909328 podStartE2EDuration="2.710909328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:22.708723764 +0000 UTC m=+1672.914305832" watchObservedRunningTime="2026-11-03 10:57:22.710909328 +0000 UTC m=+1672.916491379"
Nov 03 10:57:22.733252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.733199    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn" podStartSLOduration=2.733181958 podStartE2EDuration="2.733181958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:22.727737965 +0000 UTC m=+1672.933320016" watchObservedRunningTime="2026-11-03 10:57:22.733181958 +0000 UTC m=+1672.938764009"
Nov 03 10:57:22.747749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.747710    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8" podStartSLOduration=1.747698893 podStartE2EDuration="1.747698893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:22.742563048 +0000 UTC m=+1672.948145101" watchObservedRunningTime="2026-11-03 10:57:22.747698893 +0000 UTC m=+1672.953280943"
Nov 03 10:57:22.809876 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:22.809846    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:23.669528 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:23.669492    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:24.557555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:24.557362    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:25.295849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:25.295803    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:25.949678 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:25.949619    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:26.689638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:26.682432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:27.347207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:27.347168    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:28.153250 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:28.150054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:28.745132 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:28.737198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:29.364074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:29.364046    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:29.488418 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:29.488375    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:29.678916 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:29.678889    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:57:30.372964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:30.372926    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:31.258650 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:31.257474    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:31.455596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:31.448646    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:31.590799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:31.590720    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:31.650874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:31.650841    5817 scope.go:122] "RemoveContainer" containerID="083bb9b07b4ebf000df0979a3ff9772b143a77218ba8a874be4b5e723c160cb0"
Nov 03 10:57:31.667303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:31.666671    5817 scope.go:122] "RemoveContainer" containerID="c6ccfa073d02d5b9f6e506307c4d030101e045e5b77df6419179c680e56f525d"
Nov 03 10:57:31.680534 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:31.680512    5817 scope.go:122] "RemoveContainer" containerID="62a11d5f393d8f7cf9072138ad80d1bd00c6745ee26c46656d02ef70a4bf44a9"
Nov 03 10:57:31.729060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:31.728991    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:32.901675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:32.900671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:33.470261 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:33.470224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:33.619597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:33.618971    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:33.722651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:33.721467    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 10:57:33.722651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:33.721549    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 10:57:33.722651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:33.721809    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:57:34.459056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.457847    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:34.826343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.825695    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:34.847392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.847359    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"]
Nov 03 10:57:34.854332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.854307    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:34.860974 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.860943    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-836589\""
Nov 03 10:57:34.866196 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.866177    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-836589-dockercfg-wr8cc\""
Nov 03 10:57:34.869986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.867894    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"]
Nov 03 10:57:34.926598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.925634    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"]
Nov 03 10:57:34.976893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.970993    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7155fb21-319e-47f0-8680-359ca1a613b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl\" (UID: \"7155fb21-319e-47f0-8680-359ca1a613b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:34.976893 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:34.971111    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzdt\" (UniqueName: \"kubernetes.io/projected/7155fb21-319e-47f0-8680-359ca1a613b0-kube-api-access-cwzdt\") pod \"payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl\" (UID: \"7155fb21-319e-47f0-8680-359ca1a613b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:35.234765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.204375    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:35.234765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.229996    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7155fb21-319e-47f0-8680-359ca1a613b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl\" (UID: \"7155fb21-319e-47f0-8680-359ca1a613b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:35.234765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.230159    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzdt\" (UniqueName: \"kubernetes.io/projected/7155fb21-319e-47f0-8680-359ca1a613b0-kube-api-access-cwzdt\") pod \"payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl\" (UID: \"7155fb21-319e-47f0-8680-359ca1a613b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:35.234765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.231908    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7155fb21-319e-47f0-8680-359ca1a613b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl\" (UID: \"7155fb21-319e-47f0-8680-359ca1a613b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:35.234765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.232939    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:35.270367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.259405    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"]
Nov 03 10:57:35.276350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.276215    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzdt\" (UniqueName: \"kubernetes.io/projected/7155fb21-319e-47f0-8680-359ca1a613b0-kube-api-access-cwzdt\") pod \"payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl\" (UID: \"7155fb21-319e-47f0-8680-359ca1a613b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:35.433688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.433649    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t\" (UID: \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:35.433911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.433892    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9tfv\" (UniqueName: \"kubernetes.io/projected/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-kube-api-access-l9tfv\") pod \"payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t\" (UID: \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:35.479644 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.479598    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:35.536059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.535563    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9tfv\" (UniqueName: \"kubernetes.io/projected/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-kube-api-access-l9tfv\") pod \"payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t\" (UID: \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:35.536059 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.535837    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t\" (UID: \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:35.536746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.536719    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t\" (UID: \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:35.546456 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.546417    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9tfv\" (UniqueName: \"kubernetes.io/projected/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-kube-api-access-l9tfv\") pod \"payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t\" (UID: \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:35.610743 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.574804    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:35.826948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.826917    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"]
Nov 03 10:57:35.880972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.880902    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl" event={"ID":"7155fb21-319e-47f0-8680-359ca1a613b0","Type":"ContainerStarted","Data":"4a928db28580989f07e6d4f30fc35e2f5209a91a3e0b79a5f1a70cc0cbcf26c0"}
Nov 03 10:57:35.928238 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.928189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:35.979020 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:35.976258    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"]
Nov 03 10:57:36.921462 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:36.921426    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t" event={"ID":"f4d72517-c9c3-4c7c-a75d-86b42d6c644f","Type":"ContainerStarted","Data":"dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa"}
Nov 03 10:57:36.922091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:36.922056    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:36.922272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:36.922256    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t" event={"ID":"f4d72517-c9c3-4c7c-a75d-86b42d6c644f","Type":"ContainerStarted","Data":"8c6d8c602f18114bd9c27c9deab9ed55603d6d68c94e18824a1e728fedae64fa"}
Nov 03 10:57:36.923687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:36.923663    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl" event={"ID":"7155fb21-319e-47f0-8680-359ca1a613b0","Type":"ContainerStarted","Data":"1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a"}
Nov 03 10:57:36.923775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:36.923727    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:36.945624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:36.943753    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t" podStartSLOduration=2.943734407 podStartE2EDuration="2.943734407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:36.937349837 +0000 UTC m=+1687.142931912" watchObservedRunningTime="2026-11-03 10:57:36.943734407 +0000 UTC m=+1687.149316464"
Nov 03 10:57:36.960285 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:36.957622    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:36.967621 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:36.966646    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl" podStartSLOduration=2.966627881 podStartE2EDuration="2.966627881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:36.964321424 +0000 UTC m=+1687.169903477" watchObservedRunningTime="2026-11-03 10:57:36.966627881 +0000 UTC m=+1687.172209935"
Nov 03 10:57:37.688597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:37.682436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:38.474874 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:38.474547    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:39.208701 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:39.208191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:39.969798 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:39.968241    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:40.888600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:40.884233    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:41.056714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.056680    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"]
Nov 03 10:57:41.057214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.056927    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy" containerID="cri-o://4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e" gracePeriod=30
Nov 03 10:57:41.064957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.064930    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"]
Nov 03 10:57:41.065156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.065116    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy" containerID="cri-o://5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74" gracePeriod=30
Nov 03 10:57:41.556596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.555699    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:41.977836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.977808    5817 generic.go:356] "Generic (PLEG): container finished" podID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerID="bcc018edb94040e6656fe04c887ee40e4e495f3716cfe0104fa67b13dbab3008" exitCode=0
Nov 03 10:57:41.978094 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.977883    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" event={"ID":"76c730b7-3e59-4d9b-a03f-ff1fab06aaed","Type":"ContainerDied","Data":"bcc018edb94040e6656fe04c887ee40e4e495f3716cfe0104fa67b13dbab3008"}
Nov 03 10:57:41.978320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.978303    5817 scope.go:122] "RemoveContainer" containerID="bcc018edb94040e6656fe04c887ee40e4e495f3716cfe0104fa67b13dbab3008"
Nov 03 10:57:41.979821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.979795    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:41.979938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.979863    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:41.979938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:41.979895    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:42.255131 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:42.253453    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:42.732826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:42.732791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:42.975600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:42.969982    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:43.070652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:43.070513    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" event={"ID":"76c730b7-3e59-4d9b-a03f-ff1fab06aaed","Type":"ContainerStarted","Data":"629c94b0f55c5ff4e3329e753d37344725d16059f939fbd63db25f58a0afbf83"}
Nov 03 10:57:43.947892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:43.947854    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:43.971877 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:43.971849    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:57:43.975693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:43.975662    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 10:57:43.975816 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:43.975733    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 10:57:44.559159 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:44.556537    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:44.971029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:44.970992    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 10:57:44.971411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:44.971048    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 10:57:45.144596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:45.144296    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:45.977589 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:45.977517    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 10:57:45.978091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:45.977630    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 10:57:46.264952 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.264923    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:46.431057 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.430595    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:46.535143 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.535113    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:57:46.580034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.579999    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:46.660892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.660872    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:57:46.722113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722071    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-token\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-token\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.722318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722220    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-kube-api-access-mpjz5\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-kube-api-access-mpjz5\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.722318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722280    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-credential-socket\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.722429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722381    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-data\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.722429 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722411    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-socket\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.722537 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722504    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-certs\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.722626 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722606    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3acd137a-8e3c-4b9e-a863-73764a7f6990-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acd137a-8e3c-4b9e-a863-73764a7f6990-istiod-ca-cert\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.722689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722641    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-envoy\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.722829 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.722698    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-podinfo\") pod \"3acd137a-8e3c-4b9e-a863-73764a7f6990\" (UID: \"3acd137a-8e3c-4b9e-a863-73764a7f6990\") "
Nov 03 10:57:46.731631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.729371    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-token" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:57:46.734809 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.734779    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-certs" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.734981 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.734962    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-credential-socket" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.735134 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.735118    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-data" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.735284 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.735268    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-socket" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.738995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.738966    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-kube-api-access-mpjz5" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "kube-api-access-mpjz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:57:46.739155 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.739032    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-podinfo" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:57:46.746625 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.744188    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acd137a-8e3c-4b9e-a863-73764a7f6990-istiod-ca-cert" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:57:46.746894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.746862    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-envoy" pod "3acd137a-8e3c-4b9e-a863-73764a7f6990" (UID: "3acd137a-8e3c-4b9e-a863-73764a7f6990"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.830689    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-data\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.830736    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-kube-api-access-hsmtf\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-kube-api-access-hsmtf\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.830767    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-credential-socket\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.830804    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-envoy\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.830912    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-certs\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.830945    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-token\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-token\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.830984    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istiod-ca-cert\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.831054    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-podinfo\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.831091    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-socket\") pod \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\" (UID: \"fb23b8b1-e31c-4066-8e09-57f80e621e6d\") "
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.831903    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjz5\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-kube-api-access-mpjz5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.831929    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.831944    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.831958    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.831973    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.831994    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acd137a-8e3c-4b9e-a863-73764a7f6990-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.832009    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.832023    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.832036    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acd137a-8e3c-4b9e-a863-73764a7f6990-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.833426    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-certs" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.833690    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istiod-ca-cert" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.833850    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-data" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.833882    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-credential-socket" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.833978 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.833923    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-socket" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.846604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.840157    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-envoy" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:46.846604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.840943    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-podinfo" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:57:46.846604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.844906    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-kube-api-access-hsmtf" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "kube-api-access-hsmtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:57:46.846604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.845000    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-token" pod "fb23b8b1-e31c-4066-8e09-57f80e621e6d" (UID: "fb23b8b1-e31c-4066-8e09-57f80e621e6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937773    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937812    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsmtf\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-kube-api-access-hsmtf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937828    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937842    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937859    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937873    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937886    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937899    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb23b8b1-e31c-4066-8e09-57f80e621e6d-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.937939 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.937912    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb23b8b1-e31c-4066-8e09-57f80e621e6d-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:46.984597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.984089    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 10:57:46.984597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:46.984161    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 10:57:47.150285 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.150252    5817 generic.go:356] "Generic (PLEG): container finished" podID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerID="4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e" exitCode=0
Nov 03 10:57:47.151007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.150303    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"
Nov 03 10:57:47.151007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.150342    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" event={"ID":"3acd137a-8e3c-4b9e-a863-73764a7f6990","Type":"ContainerDied","Data":"4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e"}
Nov 03 10:57:47.151007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.150404    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb" event={"ID":"3acd137a-8e3c-4b9e-a863-73764a7f6990","Type":"ContainerDied","Data":"f5db00308aacc1532c4493d1a0bdf7597ba7da0287c20b29b2b04f88c72ce74a"}
Nov 03 10:57:47.151007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.150438    5817 scope.go:122] "RemoveContainer" containerID="4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e"
Nov 03 10:57:47.152976 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.152954    5817 generic.go:356] "Generic (PLEG): container finished" podID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerID="5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74" exitCode=0
Nov 03 10:57:47.153189 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.153162    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"
Nov 03 10:57:47.153270 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.153186    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" event={"ID":"fb23b8b1-e31c-4066-8e09-57f80e621e6d","Type":"ContainerDied","Data":"5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74"}
Nov 03 10:57:47.153270 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.153221    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l" event={"ID":"fb23b8b1-e31c-4066-8e09-57f80e621e6d","Type":"ContainerDied","Data":"0f994c978848b821f1703de199dfe40114749863a0a3cf787dbe08c2c02d8566"}
Nov 03 10:57:47.165079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.165012    5817 scope.go:122] "RemoveContainer" containerID="4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e"
Nov 03 10:57:47.165555 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:57:47.165533    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e\": container with ID starting with 4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e not found: ID does not exist" containerID="4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e"
Nov 03 10:57:47.165665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.165593    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e"} err="failed to get container status \"4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e\": rpc error: code = NotFound desc = could not find container \"4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e\": container with ID starting with 4758b4a69f2d12bb82b6ac6ffce35ad48c9ad520160cb2e64da373a36b542e3e not found: ID does not exist"
Nov 03 10:57:47.165665 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.165653    5817 scope.go:122] "RemoveContainer" containerID="5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74"
Nov 03 10:57:47.176957 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.176942    5817 scope.go:122] "RemoveContainer" containerID="5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74"
Nov 03 10:57:47.177292 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:57:47.177202    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74\": container with ID starting with 5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74 not found: ID does not exist" containerID="5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74"
Nov 03 10:57:47.177292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.177230    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74"} err="failed to get container status \"5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74\": rpc error: code = NotFound desc = could not find container \"5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74\": container with ID starting with 5c406b156553c4e3cf31ea6361f17538113c5af5c1bb08219e02785d36e72f74 not found: ID does not exist"
Nov 03 10:57:47.187658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.187556    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"]
Nov 03 10:57:47.219928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.219882    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7h7sb"]
Nov 03 10:57:47.226295 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.226271    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"]
Nov 03 10:57:47.230313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.230294    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-4s98l"]
Nov 03 10:57:47.318422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.318390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:47.402678 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.401660    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:57:47.513153 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.512441    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:47.718174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.718120    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:47.944031 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.942779    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" path="/var/lib/kubelet/pods/3acd137a-8e3c-4b9e-a863-73764a7f6990/volumes"
Nov 03 10:57:47.945115 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.944892    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" path="/var/lib/kubelet/pods/fb23b8b1-e31c-4066-8e09-57f80e621e6d/volumes"
Nov 03 10:57:47.946554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.946355    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 10:57:47.971334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.970860    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 10:57:47.971334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:47.970907    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 10:57:48.612213 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:48.612178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:48.950505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:48.950482    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 10:57:48.977802 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:48.976977    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:57:48.978756 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:48.978732    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 10:57:49.444462 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:49.443491    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:50.235324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:50.235292    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:50.377755 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:50.377718    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:51.173532 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:51.171466    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:51.876814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:51.874672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:52.415779 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:57:52.415727    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:57:52.551657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:52.551623    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:53.250112 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:53.247145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:53.894468 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:53.894431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:54.574208 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:54.574174    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:55.359596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:55.358193    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:56.069217 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:56.069181    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:56.757207 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:56.757169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:57.603598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:57.597054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:58.370145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:58.370105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:57:59.984290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:57:59.984260    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:01.110597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:01.106050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:02.099964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:02.099930    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:02.239657 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:02.239616    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:02.376866 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:02.369114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:03.212598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:03.209686    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:04.101592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:04.092027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:05.011248 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:05.011210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:05.852882 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:05.849803    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:06.524343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:06.524300    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:07.318099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.318064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:07.459153 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.459120    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:07.521361 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.521322    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"]
Nov 03 10:58:07.522415 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.522393    5817 cpu_manager.go:414] "containerMap: removing container" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy"
Nov 03 10:58:07.522497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.522417    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy"
Nov 03 10:58:07.522497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.522438    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy"
Nov 03 10:58:07.522497 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.522446    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy"
Nov 03 10:58:07.522651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.522638    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb23b8b1-e31c-4066-8e09-57f80e621e6d" containerName="istio-proxy"
Nov 03 10:58:07.522688 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.522655    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acd137a-8e3c-4b9e-a863-73764a7f6990" containerName="istio-proxy"
Nov 03 10:58:07.529628 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.527718    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.533425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.533396    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"]
Nov 03 10:58:07.533659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.533624    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:58:07.533772 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.533757    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:58:07.634832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.634744    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.635026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.634882    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkwc\" (UniqueName: \"kubernetes.io/projected/49acbc68-5125-4753-a9e9-b4fe6a96c358-kube-api-access-fxkwc\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.635107 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.635090    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.736912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.736861    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkwc\" (UniqueName: \"kubernetes.io/projected/49acbc68-5125-4753-a9e9-b4fe6a96c358-kube-api-access-fxkwc\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.737110 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.737092    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.737243 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.737226    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.737754 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.737733    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.740367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.740339    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.744634 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.744560    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkwc\" (UniqueName: \"kubernetes.io/projected/49acbc68-5125-4753-a9e9-b4fe6a96c358-kube-api-access-fxkwc\") pod \"maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:07.858995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:07.858968    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:08.175054 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:08.175022    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"]
Nov 03 10:58:08.179716 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:58:08.179690    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49acbc68_5125_4753_a9e9_b4fe6a96c358.slice/crio-a000f31f678fd30f8e2acfe4d48d3e1ef6b11d0206df6076a9af8c23994e8fd7 WatchSource:0}: Error finding container a000f31f678fd30f8e2acfe4d48d3e1ef6b11d0206df6076a9af8c23994e8fd7: Status 404 returned error can't find the container with id a000f31f678fd30f8e2acfe4d48d3e1ef6b11d0206df6076a9af8c23994e8fd7
Nov 03 10:58:08.334659 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:08.328465    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:08.434518 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:08.434479    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6" event={"ID":"49acbc68-5125-4753-a9e9-b4fe6a96c358","Type":"ContainerStarted","Data":"a000f31f678fd30f8e2acfe4d48d3e1ef6b11d0206df6076a9af8c23994e8fd7"}
Nov 03 10:58:09.065774 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.065622    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:09.445814 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.445738    5817 generic.go:356] "Generic (PLEG): container finished" podID="49acbc68-5125-4753-a9e9-b4fe6a96c358" containerID="335930e1c9b3e116dd56eeaad03f84353bd2564a38a3c86d584c126b5aeda36b" exitCode=0
Nov 03 10:58:09.446228 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.445829    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6" event={"ID":"49acbc68-5125-4753-a9e9-b4fe6a96c358","Type":"ContainerDied","Data":"335930e1c9b3e116dd56eeaad03f84353bd2564a38a3c86d584c126b5aeda36b"}
Nov 03 10:58:09.447764 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.447745    5817 generic.go:356] "Generic (PLEG): container finished" podID="851221cb-59f2-424f-8c42-40104dafa17b" containerID="bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380" exitCode=0
Nov 03 10:58:09.447867 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.447821    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" event={"ID":"851221cb-59f2-424f-8c42-40104dafa17b","Type":"ContainerDied","Data":"bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380"}
Nov 03 10:58:09.448234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.448220    5817 scope.go:122] "RemoveContainer" containerID="bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380"
Nov 03 10:58:09.449902 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.449876    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:09.449988 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.449940    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:09.449988 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.449984    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:09.862801 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:09.862692    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:10.459379 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.459338    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" event={"ID":"851221cb-59f2-424f-8c42-40104dafa17b","Type":"ContainerStarted","Data":"5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4"}
Nov 03 10:58:10.635776 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.635739    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:10.760686 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.760650    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:10.921703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.921672    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49acbc68-5125-4753-a9e9-b4fe6a96c358-kube-api-access-fxkwc\" (UniqueName: \"kubernetes.io/projected/49acbc68-5125-4753-a9e9-b4fe6a96c358-kube-api-access-fxkwc\") pod \"49acbc68-5125-4753-a9e9-b4fe6a96c358\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") "
Nov 03 10:58:10.921950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.921915    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-openshift-ca\") pod \"49acbc68-5125-4753-a9e9-b4fe6a96c358\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") "
Nov 03 10:58:10.922025 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.921965    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-xks-ca\") pod \"49acbc68-5125-4753-a9e9-b4fe6a96c358\" (UID: \"49acbc68-5125-4753-a9e9-b4fe6a96c358\") "
Nov 03 10:58:10.922356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.922327    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-openshift-ca" pod "49acbc68-5125-4753-a9e9-b4fe6a96c358" (UID: "49acbc68-5125-4753-a9e9-b4fe6a96c358"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:58:10.922744 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.922721    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:10.924267 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.924240    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49acbc68-5125-4753-a9e9-b4fe6a96c358-kube-api-access-fxkwc" pod "49acbc68-5125-4753-a9e9-b4fe6a96c358" (UID: "49acbc68-5125-4753-a9e9-b4fe6a96c358"). InnerVolumeSpecName "kube-api-access-fxkwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:58:10.924402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:10.924364    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-xks-ca" pod "49acbc68-5125-4753-a9e9-b4fe6a96c358" (UID: "49acbc68-5125-4753-a9e9-b4fe6a96c358"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:58:11.029109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.029013    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49acbc68-5125-4753-a9e9-b4fe6a96c358-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:11.029109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.029054    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxkwc\" (UniqueName: \"kubernetes.io/projected/49acbc68-5125-4753-a9e9-b4fe6a96c358-kube-api-access-fxkwc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:11.277872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.277825    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:11.453216 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.453187    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:58:11.453949 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.453930    5817 patch_prober.go:29] interesting pod/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 10:58:11.454039 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.453991    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 10:58:11.474608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.474562    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"
Nov 03 10:58:11.475063 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.474554    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6" event={"ID":"49acbc68-5125-4753-a9e9-b4fe6a96c358","Type":"ContainerDied","Data":"a000f31f678fd30f8e2acfe4d48d3e1ef6b11d0206df6076a9af8c23994e8fd7"}
Nov 03 10:58:11.475063 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:11.474806    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a000f31f678fd30f8e2acfe4d48d3e1ef6b11d0206df6076a9af8c23994e8fd7"
Nov 03 10:58:12.061745 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:12.060878    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:12.453718 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:12.453684    5817 patch_prober.go:29] interesting pod/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 10:58:12.453884 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:12.453737    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 10:58:12.961402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:12.961022    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:13.452970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:13.452941    5817 patch_prober.go:29] interesting pod/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 10:58:13.453156 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:13.452996    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 10:58:13.815485 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:13.815359    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:13.953564 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:13.953521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:14.453026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:14.452992    5817 patch_prober.go:29] interesting pod/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 10:58:14.453397 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:14.453061    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 10:58:14.868018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:14.867919    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:15.453424 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:15.453382    5817 patch_prober.go:29] interesting pod/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 10:58:15.453974 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:15.453443    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 10:58:15.697799 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:15.697763    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:16.457930 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:16.457897    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:58:16.458425 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:16.458411    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:58:16.459256 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:16.459234    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:58:16.718593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:16.716816    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:17.188335 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.188301    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"]
Nov 03 10:58:17.189392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.189364    5817 cpu_manager.go:414] "containerMap: removing container" podUID="49acbc68-5125-4753-a9e9-b4fe6a96c358" containerName="revoke-keys"
Nov 03 10:58:17.189554 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.189540    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49acbc68-5125-4753-a9e9-b4fe6a96c358" containerName="revoke-keys"
Nov 03 10:58:17.189902 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.189886    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="49acbc68-5125-4753-a9e9-b4fe6a96c358" containerName="revoke-keys"
Nov 03 10:58:17.195197 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.195174    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.197875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.197851    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"]
Nov 03 10:58:17.204620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.204600    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e11de4\"/\"kube-root-ca.crt\""
Nov 03 10:58:17.204984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.204965    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e11de4\"/\"openshift-service-ca.crt\""
Nov 03 10:58:17.209318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.208782    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e11de4\"/\"default-dockercfg-ct7wf\""
Nov 03 10:58:17.209318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.208863    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e11de4\"/\"auth-test-model-e11de4-kserve-self-signed-certs\""
Nov 03 10:58:17.260313 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.260284    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-model-cache\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.260455 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.260324    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-dshm\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.260505 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.260465    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568dfe2b-947c-446f-9eea-3a4262876815-tls-certs\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.260548 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.260535    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-home\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.260675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.260653    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqxrf\" (UniqueName: \"kubernetes.io/projected/568dfe2b-947c-446f-9eea-3a4262876815-kube-api-access-gqxrf\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.260838 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.260820    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-tmp-dir\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.263442 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.263415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.377478    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-model-cache\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.377535    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-dshm\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.377651    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568dfe2b-947c-446f-9eea-3a4262876815-tls-certs\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.377738    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-home\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.377796    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqxrf\" (UniqueName: \"kubernetes.io/projected/568dfe2b-947c-446f-9eea-3a4262876815-kube-api-access-gqxrf\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.377916    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-tmp-dir\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.378435    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-tmp-dir\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.378699    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-model-cache\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.379338    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-home\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.387624 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.386844    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-dshm\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.402606 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.401257    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568dfe2b-947c-446f-9eea-3a4262876815-tls-certs\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.415595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.404931    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqxrf\" (UniqueName: \"kubernetes.io/projected/568dfe2b-947c-446f-9eea-3a4262876815-kube-api-access-gqxrf\") pod \"auth-test-model-e11de4-kserve-6866b8749-j7cj8\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") " pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.514292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.514211    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:17.691730 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.691072    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"]
Nov 03 10:58:17.792221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.792139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:17.829809 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:58:17.829777    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod568dfe2b_947c_446f_9eea_3a4262876815.slice/crio-3285c073a0d5c91bbbdc2d39a2ea901376379b7b7172695175b0d0296862cdf5 WatchSource:0}: Error finding container 3285c073a0d5c91bbbdc2d39a2ea901376379b7b7172695175b0d0296862cdf5: Status 404 returned error can't find the container with id 3285c073a0d5c91bbbdc2d39a2ea901376379b7b7172695175b0d0296862cdf5
Nov 03 10:58:17.830345 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.830321    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"]
Nov 03 10:58:17.928225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:17.928202    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:18.596611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:18.596555    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8" event={"ID":"568dfe2b-947c-446f-9eea-3a4262876815","Type":"ContainerStarted","Data":"0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9"}
Nov 03 10:58:18.596611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:18.596607    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8" event={"ID":"568dfe2b-947c-446f-9eea-3a4262876815","Type":"ContainerStarted","Data":"3285c073a0d5c91bbbdc2d39a2ea901376379b7b7172695175b0d0296862cdf5"}
Nov 03 10:58:18.616596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:18.616514    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8" podStartSLOduration=1.616499793 podStartE2EDuration="1.616499793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:58:18.61394665 +0000 UTC m=+1728.819528707" watchObservedRunningTime="2026-11-03 10:58:18.616499793 +0000 UTC m=+1728.822081850"
Nov 03 10:58:18.646662 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:18.646628    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:19.023765 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:19.023732    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"]
Nov 03 10:58:19.028244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:19.028226    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e41c7c-npm6h"]
Nov 03 10:58:19.429336 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:19.423455    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:19.514348 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:19.514321    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:19.904830 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:19.904751    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9" path="/var/lib/kubelet/pods/ffebe9ce-8cac-48b8-b3bb-fd6d75ead2c9/volumes"
Nov 03 10:58:20.160512 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:20.160407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:20.857093 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:20.857047    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:21.666729 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:21.662972    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:22.414359 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:58:22.414322    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:58:22.489174 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:22.489123    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:22.691993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:22.691954    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8" podUID="7e17c72e-2eec-4714-841f-2763a09d054b" containerName="payload-pre-processing" containerID="cri-o://1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457" gracePeriod=25
Nov 03 10:58:22.976795 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:22.976771    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:58:23.054017 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.053984    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:23.088592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.081823    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e17c72e-2eec-4714-841f-2763a09d054b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e17c72e-2eec-4714-841f-2763a09d054b-plugins-config-volume\") pod \"7e17c72e-2eec-4714-841f-2763a09d054b\" (UID: \"7e17c72e-2eec-4714-841f-2763a09d054b\") "
Nov 03 10:58:23.088592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.081961    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e17c72e-2eec-4714-841f-2763a09d054b-kube-api-access-sw65v\" (UniqueName: \"kubernetes.io/projected/7e17c72e-2eec-4714-841f-2763a09d054b-kube-api-access-sw65v\") pod \"7e17c72e-2eec-4714-841f-2763a09d054b\" (UID: \"7e17c72e-2eec-4714-841f-2763a09d054b\") "
Nov 03 10:58:23.088592 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.088272    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e17c72e-2eec-4714-841f-2763a09d054b-plugins-config-volume" pod "7e17c72e-2eec-4714-841f-2763a09d054b" (UID: "7e17c72e-2eec-4714-841f-2763a09d054b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:58:23.095148 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.095112    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e17c72e-2eec-4714-841f-2763a09d054b-kube-api-access-sw65v" pod "7e17c72e-2eec-4714-841f-2763a09d054b" (UID: "7e17c72e-2eec-4714-841f-2763a09d054b"). InnerVolumeSpecName "kube-api-access-sw65v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:58:23.184529 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.184491    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e17c72e-2eec-4714-841f-2763a09d054b-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:23.184529 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.184520    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw65v\" (UniqueName: \"kubernetes.io/projected/7e17c72e-2eec-4714-841f-2763a09d054b-kube-api-access-sw65v\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:23.568861 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.568826    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:23.650046 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.650019    5817 generic.go:356] "Generic (PLEG): container finished" podID="7e17c72e-2eec-4714-841f-2763a09d054b" containerID="1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457" exitCode=0
Nov 03 10:58:23.650227 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.650070    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8" event={"ID":"7e17c72e-2eec-4714-841f-2763a09d054b","Type":"ContainerDied","Data":"1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457"}
Nov 03 10:58:23.650227 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.650097    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"
Nov 03 10:58:23.650227 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.650134    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8" event={"ID":"7e17c72e-2eec-4714-841f-2763a09d054b","Type":"ContainerDied","Data":"e1338d5bd9d258fb247cc99c262f56136024bafce2247931b805fb1325976e7c"}
Nov 03 10:58:23.650227 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.650158    5817 scope.go:122] "RemoveContainer" containerID="1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457"
Nov 03 10:58:23.667457 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.666556    5817 scope.go:122] "RemoveContainer" containerID="1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457"
Nov 03 10:58:23.667812 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:58:23.667791    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457\": container with ID starting with 1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457 not found: ID does not exist" containerID="1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457"
Nov 03 10:58:23.667901 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.667819    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457"} err="failed to get container status \"1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457\": rpc error: code = NotFound desc = could not find container \"1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457\": container with ID starting with 1d549b76ac6796eef4f3b796f041c833609812fc093776726ee87759b523d457 not found: ID does not exist"
Nov 03 10:58:23.682689 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.682664    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"]
Nov 03 10:58:23.687116 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.687099    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ba121-7bc9c97f9-84pq8"]
Nov 03 10:58:23.710722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.710696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:23.719693 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.719671    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"]
Nov 03 10:58:23.727687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.727665    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3ba121-4w8x6"]
Nov 03 10:58:23.848168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.848088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:23.940594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.940304    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49acbc68-5125-4753-a9e9-b4fe6a96c358" path="/var/lib/kubelet/pods/49acbc68-5125-4753-a9e9-b4fe6a96c358/volumes"
Nov 03 10:58:23.953154 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:23.953119    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e17c72e-2eec-4714-841f-2763a09d054b" path="/var/lib/kubelet/pods/7e17c72e-2eec-4714-841f-2763a09d054b/volumes"
Nov 03 10:58:24.293206 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:24.293157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:24.429595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:24.429203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:25.038598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:25.036550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:25.761598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:25.756845    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:26.376555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:26.376520    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:27.073380 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:27.071206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:27.515343 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:27.515308    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:27.528635 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:27.528613    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:27.688648 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:27.681290    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:27.721163 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:27.721137    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 10:58:28.372536 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:28.372438    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:28.616272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:28.616242    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:58:29.295414 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:29.295384    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:30.155619 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:30.155525    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:30.512230 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:30.512195    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:30.651842 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:30.651805    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:30.767482 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:30.767396    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"]
Nov 03 10:58:30.767716 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:30.767680    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy" containerID="cri-o://5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4" gracePeriod=30
Nov 03 10:58:31.460898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.460863    5817 patch_prober.go:29] interesting pod/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:58:31.461413 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.460934    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:58:31.761539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.761461    5817 scope.go:122] "RemoveContainer" containerID="fee993e3d2880683f45f5f61b53c380549f1a87fc77dc5204edb99b8f63c4104"
Nov 03 10:58:31.772938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.772720    5817 scope.go:122] "RemoveContainer" containerID="f8ddd593ea37130828121ac14f73a990808398be308542d459705a7221d4970d"
Nov 03 10:58:31.782645 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.782628    5817 scope.go:122] "RemoveContainer" containerID="4ac2651f268b43ad272a88b6c75660d3e3f3fffa1e9478b07ae2bc8a03df4dfb"
Nov 03 10:58:31.792448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.792430    5817 scope.go:122] "RemoveContainer" containerID="ae724f59885c04fee7338c2b96cb4a4315b4c8074279d068a9ff40730af979cc"
Nov 03 10:58:31.802074 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.802043    5817 scope.go:122] "RemoveContainer" containerID="685229cffa35ab3a881b6f8ee50d573bd1adbab6ad632732d55f3ed654b6221d"
Nov 03 10:58:31.912694 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.912669    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"]
Nov 03 10:58:31.913672 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.913652    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7e17c72e-2eec-4714-841f-2763a09d054b" containerName="payload-pre-processing"
Nov 03 10:58:31.913726 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.913675    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e17c72e-2eec-4714-841f-2763a09d054b" containerName="payload-pre-processing"
Nov 03 10:58:31.913912 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.913900    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e17c72e-2eec-4714-841f-2763a09d054b" containerName="payload-pre-processing"
Nov 03 10:58:31.918283 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.918262    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:31.935790 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.935743    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-b5a832-openshift-default-dockercfg-9h2vp\""
Nov 03 10:58:31.940620 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:31.940489    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"]
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.137670    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-credential-socket\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.137813    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-envoy\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.137895    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qftgn\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-kube-api-access-qftgn\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.137929    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-data\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.137964    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-token\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.138023    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-certs\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.138049    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-podinfo\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.138161    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-socket\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.143652 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.138185    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istiod-ca-cert\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.241722 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.241686    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-credential-socket\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.242125 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.242086    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-envoy\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.242310 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.242161    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-credential-socket\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.242418 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.242301    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qftgn\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-kube-api-access-qftgn\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.242537 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.242522    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-data\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.242787 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.242769    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-token\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.242965 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.242949    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-certs\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.243179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.243159    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-podinfo\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.243436 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.243416    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-socket\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.243563 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.243549    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istiod-ca-cert\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.243872 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.242971    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-data\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.244240 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.243464    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-certs\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.244369 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.244353    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-socket\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.245292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.245216    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istiod-ca-cert\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.245774 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.245748    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-envoy\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.246559 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.246534    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-podinfo\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.248503 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.248477    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:32.251020 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.251001    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qftgn\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-kube-api-access-qftgn\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.251113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.251077    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-token\") pod \"e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") " pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.539488 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.539454    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:32.791117 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.791015    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:32.829787 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.829760    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"]
Nov 03 10:58:32.834338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.833692    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:32.834338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.833801    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:32.834338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:32.833846    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:33.410604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:33.406039    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:33.768320 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:33.768281    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt" event={"ID":"e76dbae1-e1eb-4909-b36d-9adb4637bb67","Type":"ContainerStarted","Data":"004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0"}
Nov 03 10:58:33.768806 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:33.768368    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt" event={"ID":"e76dbae1-e1eb-4909-b36d-9adb4637bb67","Type":"ContainerStarted","Data":"6b9c78ea4b608cb95cf0d38fa817ccf40f55254fb49a507339361a5dcd7883f2"}
Nov 03 10:58:33.795168 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:33.795121    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt" podStartSLOduration=2.795107648 podStartE2EDuration="2.795107648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:58:33.788535738 +0000 UTC m=+1743.994117790" watchObservedRunningTime="2026-11-03 10:58:33.795107648 +0000 UTC m=+1744.000689698"
Nov 03 10:58:34.179506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:34.179475    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:34.336099 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:34.336021    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:34.481687 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:34.481556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:34.539929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:34.539876    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:34.547440 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:34.547410    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:34.776870 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:34.776792    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:34.777658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:34.777635    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:35.097740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:35.097651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:35.751597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:35.748106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:36.064875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.064852    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:58:36.112182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112143    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/851221cb-59f2-424f-8c42-40104dafa17b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851221cb-59f2-424f-8c42-40104dafa17b-istiod-ca-cert\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112218    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-data\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112403    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-istio-token\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-istio-token\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112480    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/851221cb-59f2-424f-8c42-40104dafa17b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851221cb-59f2-424f-8c42-40104dafa17b-istio-podinfo\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-credential-socket\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112647 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112614    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-certs\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112656    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-socket\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112718    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-kube-api-access-stbqw\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-kube-api-access-stbqw\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112769    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-envoy\") pod \"851221cb-59f2-424f-8c42-40104dafa17b\" (UID: \"851221cb-59f2-424f-8c42-40104dafa17b\") "
Nov 03 10:58:36.112821 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112557    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851221cb-59f2-424f-8c42-40104dafa17b-istiod-ca-cert" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:58:36.113036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112838    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-certs" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:36.113036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.112992    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-socket" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:36.113036 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.113028    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-credential-socket" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:36.113446 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.113423    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.113539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.113452    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.113539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.113468    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.113539 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.113482    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851221cb-59f2-424f-8c42-40104dafa17b-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.113740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.113585    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-data" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:36.116775 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.116746    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-envoy" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:36.117187 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.117161    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-kube-api-access-stbqw" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "kube-api-access-stbqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:58:36.117543 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.117519    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/851221cb-59f2-424f-8c42-40104dafa17b-istio-podinfo" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:58:36.117783 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.117764    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-istio-token" pod "851221cb-59f2-424f-8c42-40104dafa17b" (UID: "851221cb-59f2-424f-8c42-40104dafa17b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:58:36.215890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.215796    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.215890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.215836    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.215890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.215850    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851221cb-59f2-424f-8c42-40104dafa17b-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.215890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.215860    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stbqw\" (UniqueName: \"kubernetes.io/projected/851221cb-59f2-424f-8c42-40104dafa17b-kube-api-access-stbqw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.215890 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.215870    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851221cb-59f2-424f-8c42-40104dafa17b-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:36.292734 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.292650    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:36.763191 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.761146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:36.824234 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.823731    5817 generic.go:356] "Generic (PLEG): container finished" podID="851221cb-59f2-424f-8c42-40104dafa17b" containerID="5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4" exitCode=0
Nov 03 10:58:36.824435 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.824402    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"
Nov 03 10:58:36.824741 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.824716    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" event={"ID":"851221cb-59f2-424f-8c42-40104dafa17b","Type":"ContainerDied","Data":"5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4"}
Nov 03 10:58:36.824871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.824772    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl" event={"ID":"851221cb-59f2-424f-8c42-40104dafa17b","Type":"ContainerDied","Data":"36ce6f12c9632d2ced690206cbddad1912e2c8b6f77991afa42a27ea4dd91100"}
Nov 03 10:58:36.824871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.824804    5817 scope.go:122] "RemoveContainer" containerID="5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4"
Nov 03 10:58:36.847942 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.847922    5817 scope.go:122] "RemoveContainer" containerID="bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380"
Nov 03 10:58:36.857721 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.857698    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"]
Nov 03 10:58:36.861050 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.861031    5817 scope.go:122] "RemoveContainer" containerID="5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4"
Nov 03 10:58:36.861416 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:58:36.861389    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4\": container with ID starting with 5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4 not found: ID does not exist" containerID="5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4"
Nov 03 10:58:36.861477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.861428    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4"} err="failed to get container status \"5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4\": rpc error: code = NotFound desc = could not find container \"5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4\": container with ID starting with 5f45a73a67617e2cc60ac0bcfcb7ddd5c595e82a032c2a0853dfe418f36a27b4 not found: ID does not exist"
Nov 03 10:58:36.861477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.861453    5817 scope.go:122] "RemoveContainer" containerID="bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380"
Nov 03 10:58:36.861843 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:58:36.861817    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380\": container with ID starting with bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380 not found: ID does not exist" containerID="bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380"
Nov 03 10:58:36.861958 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.861853    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380"} err="failed to get container status \"bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380\": rpc error: code = NotFound desc = could not find container \"bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380\": container with ID starting with bd6445dec89070e3e00a3e60a8d50c48ae3eb2b6ca1d744750a430a0d556c380 not found: ID does not exist"
Nov 03 10:58:36.864723 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:36.864701    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-3ba121-openshift-default-5dcf7fbcbf-dtmgl"]
Nov 03 10:58:37.681357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:37.681318    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:37.904251 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:37.904212    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="851221cb-59f2-424f-8c42-40104dafa17b" path="/var/lib/kubelet/pods/851221cb-59f2-424f-8c42-40104dafa17b/volumes"
Nov 03 10:58:38.366726 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:38.366687    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:39.044307 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:39.044272    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:39.954826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:39.951338    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:40.748602 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:40.744648    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:41.455381 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:41.454002    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:42.022714 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:42.022684    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"]
Nov 03 10:58:42.026849 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:42.026827    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3b305-q2d6b"]
Nov 03 10:58:42.136448 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:42.136089    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:42.776008 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:42.775967    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:42.908762 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:42.908712    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:43.753700 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:43.753667    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:43.905218 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:43.905178    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f092b867-be96-4a2c-b01f-9dbb50e3e7cc" path="/var/lib/kubelet/pods/f092b867-be96-4a2c-b01f-9dbb50e3e7cc/volumes"
Nov 03 10:58:44.357157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:44.357120    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:45.070758 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.070720    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:45.197557 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.197525    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"]
Nov 03 10:58:45.198279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.198250    5817 cpu_manager.go:414] "containerMap: removing container" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy"
Nov 03 10:58:45.198279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.198271    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy"
Nov 03 10:58:45.198279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.198281    5817 cpu_manager.go:414] "containerMap: removing container" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy"
Nov 03 10:58:45.198523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.198286    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy"
Nov 03 10:58:45.198523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.198445    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy"
Nov 03 10:58:45.198523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.198459    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="851221cb-59f2-424f-8c42-40104dafa17b" containerName="istio-proxy"
Nov 03 10:58:45.202656 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.202631    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.216367 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.215148    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e11de4-openshift-default-dockercfg-l9lfn\""
Nov 03 10:58:45.227706 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.227676    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"]
Nov 03 10:58:45.349499 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.349384    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:45.354702 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.354674    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-istio-token\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.354831 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.354756    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-data\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.354888 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.354858    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-certs\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.354924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.354911    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc989773-c123-455c-94e3-8cc103087d84-istio-podinfo\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.354979 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.354958    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khr6r\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-kube-api-access-khr6r\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.355081 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.355062    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc989773-c123-455c-94e3-8cc103087d84-istiod-ca-cert\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.355166 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.355152    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-credential-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.355235 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.355222    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.355314 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.355301    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-envoy\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.447880 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.447843    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"]
Nov 03 10:58:45.461084 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.461018    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt" podUID="e76dbae1-e1eb-4909-b36d-9adb4637bb67" containerName="istio-proxy" containerID="cri-o://004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0" gracePeriod=30
Nov 03 10:58:45.462679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.459925    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-envoy\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.462679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.462505    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-istio-token\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.462679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.462617    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-data\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.462679 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.462672    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-certs\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.462941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.462725    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc989773-c123-455c-94e3-8cc103087d84-istio-podinfo\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.462941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.462755    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khr6r\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-kube-api-access-khr6r\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.462941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.462802    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc989773-c123-455c-94e3-8cc103087d84-istiod-ca-cert\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.462941 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.462884    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-credential-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.463225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.462968    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.463609 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.463562    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.464519 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.464495    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc989773-c123-455c-94e3-8cc103087d84-istiod-ca-cert\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.474851 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.474820    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-credential-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.475109 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.475086    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-data\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.483038 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.475726    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-certs\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.491594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.488181    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khr6r\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-kube-api-access-khr6r\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.500605 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.495611    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:45.574180 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.574146    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc989773-c123-455c-94e3-8cc103087d84-istio-podinfo\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.584786 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.582106    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-envoy\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.591762 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.591730    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-istio-token\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:45.848887 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:45.848855    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:46.145293 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:58:46.145263    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc989773_c123_455c_94e3_8cc103087d84.slice/crio-6af76f80ba2ac94b23bc05a26927b7685b24b4456a14be0c124e60fb3f401607 WatchSource:0}: Error finding container 6af76f80ba2ac94b23bc05a26927b7685b24b4456a14be0c124e60fb3f401607: Status 404 returned error can't find the container with id 6af76f80ba2ac94b23bc05a26927b7685b24b4456a14be0c124e60fb3f401607
Nov 03 10:58:46.146171 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.146149    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"]
Nov 03 10:58:46.148200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.148169    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:46.148290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.148228    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:46.148290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.148260    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:46.182266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.181129    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:46.874972 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.869233    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:46.957053 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.957017    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" event={"ID":"cc989773-c123-455c-94e3-8cc103087d84","Type":"ContainerStarted","Data":"2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b"}
Nov 03 10:58:46.957236 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.957082    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" event={"ID":"cc989773-c123-455c-94e3-8cc103087d84","Type":"ContainerStarted","Data":"6af76f80ba2ac94b23bc05a26927b7685b24b4456a14be0c124e60fb3f401607"}
Nov 03 10:58:46.987954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:46.987898    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" podStartSLOduration=1.987880624 podStartE2EDuration="1.987880624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:58:46.982895705 +0000 UTC m=+1757.188477771" watchObservedRunningTime="2026-11-03 10:58:46.987880624 +0000 UTC m=+1757.193462675"
Nov 03 10:58:47.329748 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:47.329711    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:47.461389 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:47.460478    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:47.849047 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:47.849018    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:47.850946 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:47.850919    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 10:58:47.851122 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:47.850963    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 10:58:48.026338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:48.026301    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"]
Nov 03 10:58:48.031022 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:48.030995    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2cfe356-lqdk6"]
Nov 03 10:58:48.142037 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:48.141901    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:48.850097 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:48.850062    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 10:58:48.850604 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:48.850122    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 10:58:48.913595 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:48.905922    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:49.582651 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:49.581820    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:49.779704 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:49.779667    5817 patch_prober.go:29] interesting pod/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:58:49.779855 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:49.779731    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt" podUID="e76dbae1-e1eb-4909-b36d-9adb4637bb67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:58:49.849331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:49.849249    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 10:58:49.849331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:49.849313    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 10:58:49.930944 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:49.930908    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c160ce-323b-477a-8dbf-dbab771c50c9" path="/var/lib/kubelet/pods/d8c160ce-323b-477a-8dbf-dbab771c50c9/volumes"
Nov 03 10:58:50.284857 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:50.283770    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:50.853832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:50.853356    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 10:58:50.853832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:50.853410    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 10:58:50.863924 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:50.863900    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:51.023483 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.023455    5817 generic.go:356] "Generic (PLEG): container finished" podID="e76dbae1-e1eb-4909-b36d-9adb4637bb67" containerID="004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0" exitCode=0
Nov 03 10:58:51.024244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.023661    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt" event={"ID":"e76dbae1-e1eb-4909-b36d-9adb4637bb67","Type":"ContainerDied","Data":"004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0"}
Nov 03 10:58:51.024244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.024099    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt" event={"ID":"e76dbae1-e1eb-4909-b36d-9adb4637bb67","Type":"ContainerDied","Data":"6b9c78ea4b608cb95cf0d38fa817ccf40f55254fb49a507339361a5dcd7883f2"}
Nov 03 10:58:51.024244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.024127    5817 scope.go:122] "RemoveContainer" containerID="004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0"
Nov 03 10:58:51.024244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.023899    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"
Nov 03 10:58:51.027357 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027331    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-credential-socket\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.027462 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027447    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-token\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-token\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.027517 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027485    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-socket\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.027597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-podinfo\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.027674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027661    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-envoy\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.027746 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027705    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-data\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.027800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027745    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-certs\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.027925 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027896    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-kube-api-access-qftgn\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-kube-api-access-qftgn\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.027994 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.027958    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istiod-ca-cert\") pod \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\" (UID: \"e76dbae1-e1eb-4909-b36d-9adb4637bb67\") "
Nov 03 10:58:51.028948 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.028867    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-socket" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:51.029555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.029422    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istiod-ca-cert" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:58:51.029555 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.029541    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-credential-socket" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:51.029968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.029762    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-data" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:51.029968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.029940    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-certs" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:51.031171 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.031146    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-envoy" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:51.031751 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.031717    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-kube-api-access-qftgn" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "kube-api-access-qftgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:58:51.033064 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.033035    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-podinfo" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:58:51.033199 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.033185    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-token" pod "e76dbae1-e1eb-4909-b36d-9adb4637bb67" (UID: "e76dbae1-e1eb-4909-b36d-9adb4637bb67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:58:51.041010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.040992    5817 scope.go:122] "RemoveContainer" containerID="004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0"
Nov 03 10:58:51.041333 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:58:51.041309    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0\": container with ID starting with 004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0 not found: ID does not exist" containerID="004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0"
Nov 03 10:58:51.041420 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.041339    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0"} err="failed to get container status \"004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0\": rpc error: code = NotFound desc = could not find container \"004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0\": container with ID starting with 004457c25e0a1969436adea07afb1eba05d2ab9d8bf6af1a2445dd52c3569ae0 not found: ID does not exist"
Nov 03 10:58:51.077681 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.077547    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131735    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131775    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131793    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131811    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131826    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131842    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131858    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qftgn\" (UniqueName: \"kubernetes.io/projected/e76dbae1-e1eb-4909-b36d-9adb4637bb67-kube-api-access-qftgn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131873    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e76dbae1-e1eb-4909-b36d-9adb4637bb67-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.131929 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.131887    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e76dbae1-e1eb-4909-b36d-9adb4637bb67-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:51.350811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.350782    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"]
Nov 03 10:58:51.356384 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.356360    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-b5a832-openshift-default-5b7797f6f5-7rgwt"]
Nov 03 10:58:51.861962 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.861882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:51.862318 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.862291    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 10:58:51.862513 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.862486    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 10:58:51.930622 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:51.930555    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76dbae1-e1eb-4909-b36d-9adb4637bb67" path="/var/lib/kubelet/pods/e76dbae1-e1eb-4909-b36d-9adb4637bb67/volumes"
Nov 03 10:58:52.416992 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:58:52.416949    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:58:52.557603 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:52.555682    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:52.854134 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:52.854106    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:52.854513 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:52.854487    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:52.855056 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:52.855037    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 10:58:53.276935 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:53.276899    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:53.964708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:53.964673    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:54.642290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:54.642254    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:55.236523 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:55.236487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:55.836640 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:55.836604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:55.970157 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:55.970121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:56.025294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:56.025259    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"]
Nov 03 10:58:56.029658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:56.029632    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19eea206-j5fpg"]
Nov 03 10:58:56.099082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:56.098986    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:56.572229 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:56.571675    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:57.279633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:57.278178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:57.919594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:57.919530    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:57.940852 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:57.938146    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0ef350-2202-4d4b-9af1-13c0e1a40136" path="/var/lib/kubelet/pods/7c0ef350-2202-4d4b-9af1-13c0e1a40136/volumes"
Nov 03 10:58:58.683660 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:58.683623    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:59.157324 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:59.157244    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:58:59.865479 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:58:59.865442    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:00.996358 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:00.996325    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:01.416422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:01.416321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:01.897991 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:01.897957    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:02.035740 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:02.035690    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:02.306600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:02.300349    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:02.438938 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:02.438906    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:02.998060 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:02.998017    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:03.868495 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:03.868462    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:04.600675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:04.600636    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:05.282325 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:05.276870    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:05.985331 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:05.985297    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:06.696151 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:06.696115    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:07.461989 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:07.461952    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:08.295594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.292080    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:08.432322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.432289    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:08.639800 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.639711    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"]
Nov 03 10:59:08.640850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.640824    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e76dbae1-e1eb-4909-b36d-9adb4637bb67" containerName="istio-proxy"
Nov 03 10:59:08.640850 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.640850    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76dbae1-e1eb-4909-b36d-9adb4637bb67" containerName="istio-proxy"
Nov 03 10:59:08.641066 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.641056    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76dbae1-e1eb-4909-b36d-9adb4637bb67" containerName="istio-proxy"
Nov 03 10:59:08.645646 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.645623    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.651823 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.651799    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-9c53b2-openshift-default-dockercfg-jmr5g\""
Nov 03 10:59:08.657091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.657068    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"]
Nov 03 10:59:08.723438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.723406    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779197    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istiod-ca-cert\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779342    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-token\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779435    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-envoy\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779483    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65v8t\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-kube-api-access-65v8t\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779515    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779543    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-credential-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779630    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-certs\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779689    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-data\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.784663 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.779760    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-podinfo\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882001 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.881950    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istiod-ca-cert\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.882064    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-token\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.882133    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-envoy\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.882169    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65v8t\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-kube-api-access-65v8t\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882205 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.882196    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.882221    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-credential-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.882266    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-certs\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.882312    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-data\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.882445 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.882365    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-podinfo\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.883272 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.883241    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.883525 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.883498    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-certs\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.883915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.883894    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-data\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.890674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.884441    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istiod-ca-cert\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.890674 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.886936    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-credential-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.924964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.924116    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-envoy\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.924964 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.924620    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-podinfo\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.925214 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.925020    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-token\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.941863 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.941598    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65v8t\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-kube-api-access-65v8t\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:08.965900 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:08.965873    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:09.067533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:09.067410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:09.298293 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:59:09.298246    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695ee621_b3ee_4b7f_bc8a_bab4912f99a0.slice/crio-09c3565fb8899804701d4072c55a7ce8cbbcb9a0e3c4e6c9708de384058c92d4 WatchSource:0}: Error finding container 09c3565fb8899804701d4072c55a7ce8cbbcb9a0e3c4e6c9708de384058c92d4: Status 404 returned error can't find the container with id 09c3565fb8899804701d4072c55a7ce8cbbcb9a0e3c4e6c9708de384058c92d4
Nov 03 10:59:09.298832 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:09.298672    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"]
Nov 03 10:59:09.301167 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:09.301136    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:09.301274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:09.301213    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:09.301274 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:09.301258    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:09.408332 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:09.408248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:10.044308 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:10.044261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:10.244546 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:10.244510    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" event={"ID":"695ee621-b3ee-4b7f-bc8a-bab4912f99a0","Type":"ContainerStarted","Data":"baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561"}
Nov 03 10:59:10.244732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:10.244563    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" event={"ID":"695ee621-b3ee-4b7f-bc8a-bab4912f99a0","Type":"ContainerStarted","Data":"09c3565fb8899804701d4072c55a7ce8cbbcb9a0e3c4e6c9708de384058c92d4"}
Nov 03 10:59:10.270258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:10.270204    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" podStartSLOduration=2.2701854949999998 podStartE2EDuration="2.270185495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:10.265937539 +0000 UTC m=+1780.471519590" watchObservedRunningTime="2026-11-03 10:59:10.270185495 +0000 UTC m=+1780.475767563"
Nov 03 10:59:10.829597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:10.820593    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:10.966294 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:10.966266    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:10.972752 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:10.972720    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:11.253487 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:11.253455    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:11.254338 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:11.254316    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:11.447732 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:11.447693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:12.211676 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:12.211643    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:12.652509 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:12.652416    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:12.786501 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:12.786464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:13.058085 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.058050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:13.814810 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.814772    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"]
Nov 03 10:59:13.818449 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.818426    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:13.821864 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.821838    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:13.825026 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.825000    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-9c53b2\""
Nov 03 10:59:13.835317 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.834354    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"]
Nov 03 10:59:13.841011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.840989    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"]
Nov 03 10:59:13.845507 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.845488    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:13.865411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.865381    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"]
Nov 03 10:59:13.866508 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.866470    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-9c53b2-dockercfg-xsblc\""
Nov 03 10:59:13.897432 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.897401    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c883a-f036-4060-8989-2c4ee57dc590-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq\" (UID: \"654c883a-f036-4060-8989-2c4ee57dc590\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:13.897610 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.897518    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkl7\" (UniqueName: \"kubernetes.io/projected/654c883a-f036-4060-8989-2c4ee57dc590-kube-api-access-8gkl7\") pod \"payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq\" (UID: \"654c883a-f036-4060-8989-2c4ee57dc590\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:13.897699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.897684    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4313de37-29db-4724-b33e-8783cfc1b979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l\" (UID: \"4313de37-29db-4724-b33e-8783cfc1b979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:13.897757 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.897705    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvdl\" (UniqueName: \"kubernetes.io/projected/4313de37-29db-4724-b33e-8783cfc1b979-kube-api-access-hpvdl\") pod \"payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l\" (UID: \"4313de37-29db-4724-b33e-8783cfc1b979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:14.004596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.999590    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4313de37-29db-4724-b33e-8783cfc1b979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l\" (UID: \"4313de37-29db-4724-b33e-8783cfc1b979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:14.004596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:13.999649    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvdl\" (UniqueName: \"kubernetes.io/projected/4313de37-29db-4724-b33e-8783cfc1b979-kube-api-access-hpvdl\") pod \"payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l\" (UID: \"4313de37-29db-4724-b33e-8783cfc1b979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:14.004596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.000236    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c883a-f036-4060-8989-2c4ee57dc590-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq\" (UID: \"654c883a-f036-4060-8989-2c4ee57dc590\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:14.004596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.000360    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkl7\" (UniqueName: \"kubernetes.io/projected/654c883a-f036-4060-8989-2c4ee57dc590-kube-api-access-8gkl7\") pod \"payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq\" (UID: \"654c883a-f036-4060-8989-2c4ee57dc590\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:14.004596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.002106    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4313de37-29db-4724-b33e-8783cfc1b979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l\" (UID: \"4313de37-29db-4724-b33e-8783cfc1b979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:14.030757 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.030719    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c883a-f036-4060-8989-2c4ee57dc590-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq\" (UID: \"654c883a-f036-4060-8989-2c4ee57dc590\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:14.031083 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.031051    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:14.037281 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.037225    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkl7\" (UniqueName: \"kubernetes.io/projected/654c883a-f036-4060-8989-2c4ee57dc590-kube-api-access-8gkl7\") pod \"payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq\" (UID: \"654c883a-f036-4060-8989-2c4ee57dc590\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:14.063658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.062282    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvdl\" (UniqueName: \"kubernetes.io/projected/4313de37-29db-4724-b33e-8783cfc1b979-kube-api-access-hpvdl\") pod \"payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l\" (UID: \"4313de37-29db-4724-b33e-8783cfc1b979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:14.161392 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.161308    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:14.188915 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.188884    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:14.465225 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:59:14.465192    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4313de37_29db_4724_b33e_8783cfc1b979.slice/crio-2fc95f5d5cce6fde9c2af635f76128e06af5aa74fe3f8361bdc239d52d953e97 WatchSource:0}: Error finding container 2fc95f5d5cce6fde9c2af635f76128e06af5aa74fe3f8361bdc239d52d953e97: Status 404 returned error can't find the container with id 2fc95f5d5cce6fde9c2af635f76128e06af5aa74fe3f8361bdc239d52d953e97
Nov 03 10:59:14.465683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.465662    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"]
Nov 03 10:59:14.507886 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.507859    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"]
Nov 03 10:59:14.510325 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:59:14.510301    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654c883a_f036_4060_8989_2c4ee57dc590.slice/crio-d92fe2f87fc82b758a37a107fd95fa8739846f5a908b89cc436f067ae52963bc WatchSource:0}: Error finding container d92fe2f87fc82b758a37a107fd95fa8739846f5a908b89cc436f067ae52963bc: Status 404 returned error can't find the container with id d92fe2f87fc82b758a37a107fd95fa8739846f5a908b89cc436f067ae52963bc
Nov 03 10:59:14.669476 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:14.669380    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:15.283434 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.276037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:15.303048 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.303006    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l" event={"ID":"4313de37-29db-4724-b33e-8783cfc1b979","Type":"ContainerStarted","Data":"73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524"}
Nov 03 10:59:15.303244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.303078    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l" event={"ID":"4313de37-29db-4724-b33e-8783cfc1b979","Type":"ContainerStarted","Data":"2fc95f5d5cce6fde9c2af635f76128e06af5aa74fe3f8361bdc239d52d953e97"}
Nov 03 10:59:15.303244 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.303109    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:15.305258 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.305229    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq" event={"ID":"654c883a-f036-4060-8989-2c4ee57dc590","Type":"ContainerStarted","Data":"219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387"}
Nov 03 10:59:15.305377 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.305274    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq" event={"ID":"654c883a-f036-4060-8989-2c4ee57dc590","Type":"ContainerStarted","Data":"d92fe2f87fc82b758a37a107fd95fa8739846f5a908b89cc436f067ae52963bc"}
Nov 03 10:59:15.305633 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.305616    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:15.321228 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.321173    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l" podStartSLOduration=2.321159093 podStartE2EDuration="2.321159093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:15.318735145 +0000 UTC m=+1785.524317197" watchObservedRunningTime="2026-11-03 10:59:15.321159093 +0000 UTC m=+1785.526741164"
Nov 03 10:59:15.338460 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.338375    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq" podStartSLOduration=2.338356708 podStartE2EDuration="2.338356708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:15.333687798 +0000 UTC m=+1785.539269841" watchObservedRunningTime="2026-11-03 10:59:15.338356708 +0000 UTC m=+1785.543938761"
Nov 03 10:59:15.943917 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:15.940260    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:16.664559 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:16.661192    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:17.277614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:17.274329    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:17.875597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:17.869060    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:18.022631 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:18.022555    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"]
Nov 03 10:59:18.026967 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:18.026943    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a5e72d-bk5s7"]
Nov 03 10:59:18.653000 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:18.652962    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:19.291599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:19.285436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:19.869658 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:19.864353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:19.911286 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:19.911254    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c96ddb-fb91-47f8-aa22-4ab952c86199" path="/var/lib/kubelet/pods/d0c96ddb-fb91-47f8-aa22-4ab952c86199/volumes"
Nov 03 10:59:20.459683 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:20.459637    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:21.047130 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:21.047082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:21.641292 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:21.641261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:21.770253 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:21.770217    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:21.922975 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:21.922888    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:22.023221 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:22.023185    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"]
Nov 03 10:59:22.027797 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:22.027767    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989db9cc-jmd8d"]
Nov 03 10:59:22.417003 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:59:22.416968    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:59:22.549853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:22.549753    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:23.217506 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:23.208410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:23.834594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:23.831196    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:23.906177 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:23.906140    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87486b01-289a-4295-83a0-4b00ff2e44d8" path="/var/lib/kubelet/pods/87486b01-289a-4295-83a0-4b00ff2e44d8/volumes"
Nov 03 10:59:24.585179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:24.585099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:25.185594 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:25.184853    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:25.768596 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:25.765827    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:25.884898 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:25.884868    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 10:59:26.317954 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:26.317927    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 10:59:26.318152 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:26.317994    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 10:59:26.563533 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:26.563250    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:27.446390 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:27.446328    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:28.127993 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:28.127945    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:28.854934 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:28.854884    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:29.674597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:29.666461    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:30.282471 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:30.282437    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:30.978950 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:30.978911    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:31.670040 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:31.669943    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:31.890091 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:31.890060    5817 scope.go:122] "RemoveContainer" containerID="6c20a1487c323c797dabd921bbb85ce928e2e56a12e5839375938db7800a7e91"
Nov 03 10:59:31.901851 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:31.901828    5817 scope.go:122] "RemoveContainer" containerID="2d71f598bc76c5c06b1b2c385a8e839c63d4346fe3c76c0b9bc38a58fe1d2146"
Nov 03 10:59:31.912360 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:31.912340    5817 scope.go:122] "RemoveContainer" containerID="3a8812eb621fad33c008084e6dceb88529d6fc68adbb75e4b3e261a3933671df"
Nov 03 10:59:31.921960 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:31.921925    5817 scope.go:122] "RemoveContainer" containerID="29dd3641a8388da92253de9a224605dc7ba76389ca3ebe1146bf9968ec926492"
Nov 03 10:59:32.396387 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:32.396339    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:33.571303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:33.571262    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:34.540763 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:34.540728    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:35.023476 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:35.023444    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"]
Nov 03 10:59:35.027892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:35.027865    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31a69b8-v6fkn"]
Nov 03 10:59:35.292113 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:35.292026    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:35.905749 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:35.905717    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e299c0-33d2-4e04-80a8-cd3588f5cbf2" path="/var/lib/kubelet/pods/19e299c0-33d2-4e04-80a8-cd3588f5cbf2/volumes"
Nov 03 10:59:36.121847 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:36.121646    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:36.766378 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:36.766346    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:37.382401 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:37.382365    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:38.077959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:38.077925    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:38.222871 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:38.222837    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:38.537282 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:38.537238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:38.768558 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:38.768523    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:38.937856 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:38.937815    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"]
Nov 03 10:59:38.945995 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:38.945969    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:38.949892 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:38.949870    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-ef1662-openshift-default-dockercfg-c2trg\""
Nov 03 10:59:38.954030 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:38.954006    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"]
Nov 03 10:59:39.071599 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.070606    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"]
Nov 03 10:59:39.088034 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.087971    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.119509    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-envoy\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.119583    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-credential-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.119647    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.119779    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-certs\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.119840    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8z5\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-kube-api-access-hb8z5\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.119951    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-data\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.119988    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-token\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.120024    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-podinfo\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.122496 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.120083    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759a79a3-3e94-4ac4-8435-b92d3835776c-istiod-ca-cert\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.145614 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.143937    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f773ba-8tz7g"]
Nov 03 10:59:39.222608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.221477    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-podinfo\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.223303 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.223267    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759a79a3-3e94-4ac4-8435-b92d3835776c-istiod-ca-cert\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.224984 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.224557    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-envoy\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.225182 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.225156    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-credential-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.231516 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.231496    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.232170 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.231402    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-envoy\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.232322 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.224353    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759a79a3-3e94-4ac4-8435-b92d3835776c-istiod-ca-cert\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.232437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.231121    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-credential-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.232542 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.232020    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.232826 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.232809    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-certs\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.233417 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.233270    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-certs\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.234045 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.233383    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8z5\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-kube-api-access-hb8z5\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.234350 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.234331    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-data\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.234836 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.234769    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-data\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.235011 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.234986    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-token\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.249029 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.249005    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-podinfo\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.257200 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.254023    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8z5\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-kube-api-access-hb8z5\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.272463 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.272431    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-token\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.460032 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.460000    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:39.569127 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.569033    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:39.875252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.875066    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:39.901595 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:59:39.900622    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759a79a3_3e94_4ac4_8435_b92d3835776c.slice/crio-e671d52f570dd7ad1181814aab038d746f5ed8ecd0056832d546328d48c50744 WatchSource:0}: Error finding container e671d52f570dd7ad1181814aab038d746f5ed8ecd0056832d546328d48c50744: Status 404 returned error can't find the container with id e671d52f570dd7ad1181814aab038d746f5ed8ecd0056832d546328d48c50744
Nov 03 10:59:39.907010 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.906967    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:39.907145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.907055    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:39.907145 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.907096    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:39.912853 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.912525    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f" path="/var/lib/kubelet/pods/6a8d8e8f-5687-4ad0-a65c-3b3ac40eb95f/volumes"
Nov 03 10:59:39.912928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:39.912894    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"]
Nov 03 10:59:40.550312 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:40.550272    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:40.581473 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:40.581433    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" event={"ID":"759a79a3-3e94-4ac4-8435-b92d3835776c","Type":"ContainerStarted","Data":"f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc"}
Nov 03 10:59:40.581959 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:40.581495    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" event={"ID":"759a79a3-3e94-4ac4-8435-b92d3835776c","Type":"ContainerStarted","Data":"e671d52f570dd7ad1181814aab038d746f5ed8ecd0056832d546328d48c50744"}
Nov 03 10:59:40.603891 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:40.603831    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podStartSLOduration=2.603813639 podStartE2EDuration="2.603813639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:40.600250278 +0000 UTC m=+1810.805832380" watchObservedRunningTime="2026-11-03 10:59:40.603813639 +0000 UTC m=+1810.809395692"
Nov 03 10:59:41.247593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:41.247194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:41.569422 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:41.569338    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:41.575904 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:41.575878    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:41.593437 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:41.593406    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:41.594266 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:41.594248    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 10:59:41.882708 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:41.877120    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:42.680695 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:42.680658    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:42.829279 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:42.829246    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:43.095675 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:43.095549    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:43.238457 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:43.238411    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:43.465611 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:43.464301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:44.081202 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.078624    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:44.354079 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.353991    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:44.442179 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.442101    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"]
Nov 03 10:59:44.452894 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.452859    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:44.458225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.458082    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-ef1662-dockercfg-csbgj\""
Nov 03 10:59:44.458225 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.458197    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-ef1662\""
Nov 03 10:59:44.470041 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.470017    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"]
Nov 03 10:59:44.572438 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.545529    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:44.587630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.573815    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5bs\" (UniqueName: \"kubernetes.io/projected/dd560ff4-8e15-492e-9278-c636496dd2be-kube-api-access-5p5bs\") pod \"payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh\" (UID: \"dd560ff4-8e15-492e-9278-c636496dd2be\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:44.587630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.573910    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd560ff4-8e15-492e-9278-c636496dd2be-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh\" (UID: \"dd560ff4-8e15-492e-9278-c636496dd2be\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:44.587630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.577627    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"]
Nov 03 10:59:44.685803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.677712    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5bs\" (UniqueName: \"kubernetes.io/projected/dd560ff4-8e15-492e-9278-c636496dd2be-kube-api-access-5p5bs\") pod \"payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh\" (UID: \"dd560ff4-8e15-492e-9278-c636496dd2be\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:44.685803 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.677806    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd560ff4-8e15-492e-9278-c636496dd2be-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh\" (UID: \"dd560ff4-8e15-492e-9278-c636496dd2be\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:44.688171 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.688098    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd560ff4-8e15-492e-9278-c636496dd2be-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh\" (UID: \"dd560ff4-8e15-492e-9278-c636496dd2be\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:44.735999 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.735949    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5bs\" (UniqueName: \"kubernetes.io/projected/dd560ff4-8e15-492e-9278-c636496dd2be-kube-api-access-5p5bs\") pod \"payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh\" (UID: \"dd560ff4-8e15-492e-9278-c636496dd2be\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:44.769844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.749899    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"]
Nov 03 10:59:44.769844 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.750030    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:44.881102 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.881068    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7kf2\" (UniqueName: \"kubernetes.io/projected/c6737167-e4c9-4287-bd9e-9151f1f14eed-kube-api-access-g7kf2\") pod \"payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt\" (UID: \"c6737167-e4c9-4287-bd9e-9151f1f14eed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:44.881330 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.881169    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6737167-e4c9-4287-bd9e-9151f1f14eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt\" (UID: \"c6737167-e4c9-4287-bd9e-9151f1f14eed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:44.982334 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.982259    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6737167-e4c9-4287-bd9e-9151f1f14eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt\" (UID: \"c6737167-e4c9-4287-bd9e-9151f1f14eed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:44.982489 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.982473    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7kf2\" (UniqueName: \"kubernetes.io/projected/c6737167-e4c9-4287-bd9e-9151f1f14eed-kube-api-access-g7kf2\") pod \"payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt\" (UID: \"c6737167-e4c9-4287-bd9e-9151f1f14eed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:44.983082 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.983062    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6737167-e4c9-4287-bd9e-9151f1f14eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt\" (UID: \"c6737167-e4c9-4287-bd9e-9151f1f14eed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:44.990699 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:44.990675    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7kf2\" (UniqueName: \"kubernetes.io/projected/c6737167-e4c9-4287-bd9e-9151f1f14eed-kube-api-access-g7kf2\") pod \"payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt\" (UID: \"c6737167-e4c9-4287-bd9e-9151f1f14eed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:45.007703 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.007682    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:45.096712 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.096681    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:45.308124 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.308093    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:45.309014 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:59:45.308990    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd560ff4_8e15_492e_9278_c636496dd2be.slice/crio-3be503fd6003d46460f8ab436dab6c86709dfc01f33d6c440b2271aa613ae8d2 WatchSource:0}: Error finding container 3be503fd6003d46460f8ab436dab6c86709dfc01f33d6c440b2271aa613ae8d2: Status 404 returned error can't find the container with id 3be503fd6003d46460f8ab436dab6c86709dfc01f33d6c440b2271aa613ae8d2
Nov 03 10:59:45.310411 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.310391    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"]
Nov 03 10:59:45.474471 ip-10-0-138-233 kubenswrapper[5817]: W1103 10:59:45.474442    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6737167_e4c9_4287_bd9e_9151f1f14eed.slice/crio-9044904adc94ffe079d5494346f8497a6401cb6184f505cb42a88b61f11292f1 WatchSource:0}: Error finding container 9044904adc94ffe079d5494346f8497a6401cb6184f505cb42a88b61f11292f1: Status 404 returned error can't find the container with id 9044904adc94ffe079d5494346f8497a6401cb6184f505cb42a88b61f11292f1
Nov 03 10:59:45.474690 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.474667    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"]
Nov 03 10:59:45.764638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.764095    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh" event={"ID":"dd560ff4-8e15-492e-9278-c636496dd2be","Type":"ContainerStarted","Data":"27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe"}
Nov 03 10:59:45.764638 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.764146    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh" event={"ID":"dd560ff4-8e15-492e-9278-c636496dd2be","Type":"ContainerStarted","Data":"3be503fd6003d46460f8ab436dab6c86709dfc01f33d6c440b2271aa613ae8d2"}
Nov 03 10:59:45.764970 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.764671    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:45.767477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.766819    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt" event={"ID":"c6737167-e4c9-4287-bd9e-9151f1f14eed","Type":"ContainerStarted","Data":"34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470"}
Nov 03 10:59:45.767477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.766855    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt" event={"ID":"c6737167-e4c9-4287-bd9e-9151f1f14eed","Type":"ContainerStarted","Data":"9044904adc94ffe079d5494346f8497a6401cb6184f505cb42a88b61f11292f1"}
Nov 03 10:59:45.767477 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.767075    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:45.793000 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.792939    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh" podStartSLOduration=1.7929199759999999 podStartE2EDuration="1.792919976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:45.790847001 +0000 UTC m=+1815.996429053" watchObservedRunningTime="2026-11-03 10:59:45.792919976 +0000 UTC m=+1815.998502026"
Nov 03 10:59:45.809851 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:45.809784    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt" podStartSLOduration=1.809764397 podStartE2EDuration="1.809764397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:45.80932268 +0000 UTC m=+1816.014904730" watchObservedRunningTime="2026-11-03 10:59:45.809764397 +0000 UTC m=+1816.015346449"
Nov 03 10:59:46.135806 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:46.135671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:46.748007 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:46.746546    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:47.365288 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:47.365256    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:47.510598 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:47.501589    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:47.976600 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:47.967272    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:48.062280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:48.061927    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"]
Nov 03 10:59:48.062280 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:48.061974    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faeb35e0-8j5mk"]
Nov 03 10:59:48.681597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:48.674959    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:49.255264 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:49.248918    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:49.860423 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:49.860385    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:49.946021 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:49.945990    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a102aa71-6f66-4838-8172-ba70d5262a28" path="/var/lib/kubelet/pods/a102aa71-6f66-4838-8172-ba70d5262a28/volumes"
Nov 03 10:59:50.563863 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:50.563814    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:51.256597 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:51.251117    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:51.873475 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:51.873067    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:52.417768 ip-10-0-138-233 kubenswrapper[5817]: E1103 10:59:52.417732    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 10:59:52.548875 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:52.548760    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:53.165710 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:53.165672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:53.296811 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:53.296768    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:53.438049 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:53.437994    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:54.136738 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:54.136699    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:54.770495 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:54.770419    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:55.463593 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:55.460315    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:56.179130 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:56.179029    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:56.255252 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:56.255175    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 10:59:56.255403 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:56.255244    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 10:59:56.796986 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:56.796958    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 10:59:56.797161 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:56.797039    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 10:59:56.856911 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:56.856857    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:57.003018 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:57.002991    5817 generic.go:356] "Generic (PLEG): container finished" podID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerID="baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561" exitCode=0
Nov 03 10:59:57.003202 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:57.003074    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" event={"ID":"695ee621-b3ee-4b7f-bc8a-bab4912f99a0","Type":"ContainerDied","Data":"baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561"}
Nov 03 10:59:57.003608 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:57.003558    5817 scope.go:122] "RemoveContainer" containerID="baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561"
Nov 03 10:59:57.005356 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:57.005328    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:57.005441 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:57.005400    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:57.005441 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:57.005433    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:57.542290 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:57.542255    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:58.029630 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:58.029554    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" event={"ID":"695ee621-b3ee-4b7f-bc8a-bab4912f99a0","Type":"ContainerStarted","Data":"41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467"}
Nov 03 10:59:58.282632 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:58.282485    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:58.955469 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:58.955436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:58.969591 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:58.969543    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 10:59:58.969968 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:58.969945    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 10:59:58.970128 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:58.970106    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 10:59:59.575212 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:59.575180    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 10:59:59.966402 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:59.966368    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 10:59:59.966928 ip-10-0-138-233 kubenswrapper[5817]: I1103 10:59:59.966426    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 11:00:00.137955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.137914    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"]
Nov 03 11:00:00.144487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.144461    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:00.147733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.147704    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"]
Nov 03 11:00:00.155089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.150404    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:00:00.155828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.155805    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:00.162294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.162273    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"]
Nov 03 11:00:00.169691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.169668    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:00.174278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.174258    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"]
Nov 03 11:00:00.229640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.212420    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4235c3-e211-4dae-93e7-d43181533579-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9\" (UID: \"0a4235c3-e211-4dae-93e7-d43181533579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:00.229946 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.229916    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b858d6dd-e7ac-4895-84e2-fea16754cf28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7\" (UID: \"b858d6dd-e7ac-4895-84e2-fea16754cf28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:00.230164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.230139    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs2tr\" (UniqueName: \"kubernetes.io/projected/f04f7770-fc4a-4c82-a644-42f81a284965-kube-api-access-bs2tr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk\" (UID: \"f04f7770-fc4a-4c82-a644-42f81a284965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:00.230396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.230377    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trccf\" (UniqueName: \"kubernetes.io/projected/0a4235c3-e211-4dae-93e7-d43181533579-kube-api-access-trccf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9\" (UID: \"0a4235c3-e211-4dae-93e7-d43181533579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:00.230677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.230650    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04f7770-fc4a-4c82-a644-42f81a284965-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk\" (UID: \"f04f7770-fc4a-4c82-a644-42f81a284965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:00.230920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.230904    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hxd\" (UniqueName: \"kubernetes.io/projected/b858d6dd-e7ac-4895-84e2-fea16754cf28-kube-api-access-j8hxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7\" (UID: \"b858d6dd-e7ac-4895-84e2-fea16754cf28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:00.235778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.235756    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"]
Nov 03 11:00:00.249586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.249543    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"]
Nov 03 11:00:00.323598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.317828    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"]
Nov 03 11:00:00.324883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.324858    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:00.334788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.334764    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:00.336830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.336800    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4235c3-e211-4dae-93e7-d43181533579-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9\" (UID: \"0a4235c3-e211-4dae-93e7-d43181533579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:00.336944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.336886    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b858d6dd-e7ac-4895-84e2-fea16754cf28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7\" (UID: \"b858d6dd-e7ac-4895-84e2-fea16754cf28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:00.337009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.336953    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs2tr\" (UniqueName: \"kubernetes.io/projected/f04f7770-fc4a-4c82-a644-42f81a284965-kube-api-access-bs2tr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk\" (UID: \"f04f7770-fc4a-4c82-a644-42f81a284965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:00.337062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.337025    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trccf\" (UniqueName: \"kubernetes.io/projected/0a4235c3-e211-4dae-93e7-d43181533579-kube-api-access-trccf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9\" (UID: \"0a4235c3-e211-4dae-93e7-d43181533579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:00.337151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.337133    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04f7770-fc4a-4c82-a644-42f81a284965-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk\" (UID: \"f04f7770-fc4a-4c82-a644-42f81a284965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:00.337265 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.337251    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hxd\" (UniqueName: \"kubernetes.io/projected/b858d6dd-e7ac-4895-84e2-fea16754cf28-kube-api-access-j8hxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7\" (UID: \"b858d6dd-e7ac-4895-84e2-fea16754cf28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:00.338839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.338804    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4235c3-e211-4dae-93e7-d43181533579-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9\" (UID: \"0a4235c3-e211-4dae-93e7-d43181533579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:00.339419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.339067    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b858d6dd-e7ac-4895-84e2-fea16754cf28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7\" (UID: \"b858d6dd-e7ac-4895-84e2-fea16754cf28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:00.339890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.339838    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04f7770-fc4a-4c82-a644-42f81a284965-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk\" (UID: \"f04f7770-fc4a-4c82-a644-42f81a284965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:00.348086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.348030    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs2tr\" (UniqueName: \"kubernetes.io/projected/f04f7770-fc4a-4c82-a644-42f81a284965-kube-api-access-bs2tr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk\" (UID: \"f04f7770-fc4a-4c82-a644-42f81a284965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:00.349115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.349089    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trccf\" (UniqueName: \"kubernetes.io/projected/0a4235c3-e211-4dae-93e7-d43181533579-kube-api-access-trccf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9\" (UID: \"0a4235c3-e211-4dae-93e7-d43181533579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:00.349226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.349203    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hxd\" (UniqueName: \"kubernetes.io/projected/b858d6dd-e7ac-4895-84e2-fea16754cf28-kube-api-access-j8hxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7\" (UID: \"b858d6dd-e7ac-4895-84e2-fea16754cf28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:00.352294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.352273    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"]
Nov 03 11:00:00.366866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.362697    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"]
Nov 03 11:00:00.373739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.373719    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"]
Nov 03 11:00:00.374000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.373938    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:00.376662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.376642    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"]
Nov 03 11:00:00.414609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.393932    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:00.454772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.454744    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"]
Nov 03 11:00:00.455312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.455279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:00.479637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.456376    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmprk\" (UniqueName: \"kubernetes.io/projected/b0a9bb76-42d9-444e-997b-69edb5081787-kube-api-access-dmprk\") pod \"maas-api-key-cleanup-29895060-8lw9p\" (UID: \"b0a9bb76-42d9-444e-997b-69edb5081787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:00.479637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.456458    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e9598e-3590-4926-b76d-e7e4b83fcf8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr\" (UID: \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:00.479637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.456522    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025c2900-5963-4755-9c63-6dba56744a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5\" (UID: \"025c2900-5963-4755-9c63-6dba56744a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:00.479637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.456659    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtjh5\" (UniqueName: \"kubernetes.io/projected/025c2900-5963-4755-9c63-6dba56744a43-kube-api-access-gtjh5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5\" (UID: \"025c2900-5963-4755-9c63-6dba56744a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:00.479637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.456705    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e68bec-133c-441a-a470-fb99744a0bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9\" (UID: \"b0e68bec-133c-441a-a470-fb99744a0bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:00.479637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.456758    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a9bb76-42d9-444e-997b-69edb5081787-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895060-8lw9p\" (UID: \"b0a9bb76-42d9-444e-997b-69edb5081787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:00.479637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.456827    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qgp\" (UniqueName: \"kubernetes.io/projected/b0e68bec-133c-441a-a470-fb99744a0bce-kube-api-access-47qgp\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9\" (UID: \"b0e68bec-133c-441a-a470-fb99744a0bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:00.479637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.456883    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp9sk\" (UniqueName: \"kubernetes.io/projected/21e9598e-3590-4926-b76d-e7e4b83fcf8a-kube-api-access-sp9sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr\" (UID: \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:00.519597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.512651    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"]
Nov 03 11:00:00.519597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.512764    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:00.522223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.522172    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"]
Nov 03 11:00:00.528874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.528855    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"]
Nov 03 11:00:00.535767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.535750    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:00.536952 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.536928    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"]
Nov 03 11:00:00.542137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.542119    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:00.558563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.558529    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmprk\" (UniqueName: \"kubernetes.io/projected/b0a9bb76-42d9-444e-997b-69edb5081787-kube-api-access-dmprk\") pod \"maas-api-key-cleanup-29895060-8lw9p\" (UID: \"b0a9bb76-42d9-444e-997b-69edb5081787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:00.558708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.558641    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e9598e-3590-4926-b76d-e7e4b83fcf8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr\" (UID: \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:00.558780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.558708    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025c2900-5963-4755-9c63-6dba56744a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5\" (UID: \"025c2900-5963-4755-9c63-6dba56744a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:00.558969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.558824    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtjh5\" (UniqueName: \"kubernetes.io/projected/025c2900-5963-4755-9c63-6dba56744a43-kube-api-access-gtjh5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5\" (UID: \"025c2900-5963-4755-9c63-6dba56744a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:00.558969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.558869    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e68bec-133c-441a-a470-fb99744a0bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9\" (UID: \"b0e68bec-133c-441a-a470-fb99744a0bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:00.558969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.558926    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a9bb76-42d9-444e-997b-69edb5081787-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895060-8lw9p\" (UID: \"b0a9bb76-42d9-444e-997b-69edb5081787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:00.559131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.558973    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qgp\" (UniqueName: \"kubernetes.io/projected/b0e68bec-133c-441a-a470-fb99744a0bce-kube-api-access-47qgp\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9\" (UID: \"b0e68bec-133c-441a-a470-fb99744a0bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:00.559131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.559013    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp9sk\" (UniqueName: \"kubernetes.io/projected/21e9598e-3590-4926-b76d-e7e4b83fcf8a-kube-api-access-sp9sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr\" (UID: \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:00.560108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.560079    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a9bb76-42d9-444e-997b-69edb5081787-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895060-8lw9p\" (UID: \"b0a9bb76-42d9-444e-997b-69edb5081787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:00.560210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.560080    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025c2900-5963-4755-9c63-6dba56744a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5\" (UID: \"025c2900-5963-4755-9c63-6dba56744a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:00.560210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.560188    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e9598e-3590-4926-b76d-e7e4b83fcf8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr\" (UID: \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:00.560380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.560097    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e68bec-133c-441a-a470-fb99744a0bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9\" (UID: \"b0e68bec-133c-441a-a470-fb99744a0bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:00.568000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.567969    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmprk\" (UniqueName: \"kubernetes.io/projected/b0a9bb76-42d9-444e-997b-69edb5081787-kube-api-access-dmprk\") pod \"maas-api-key-cleanup-29895060-8lw9p\" (UID: \"b0a9bb76-42d9-444e-997b-69edb5081787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:00.571374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.571351    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtjh5\" (UniqueName: \"kubernetes.io/projected/025c2900-5963-4755-9c63-6dba56744a43-kube-api-access-gtjh5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5\" (UID: \"025c2900-5963-4755-9c63-6dba56744a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:00.571507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.571422    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp9sk\" (UniqueName: \"kubernetes.io/projected/21e9598e-3590-4926-b76d-e7e4b83fcf8a-kube-api-access-sp9sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr\" (UID: \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:00.572968 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.572934    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qgp\" (UniqueName: \"kubernetes.io/projected/b0e68bec-133c-441a-a470-fb99744a0bce-kube-api-access-47qgp\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9\" (UID: \"b0e68bec-133c-441a-a470-fb99744a0bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:00.617498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.614861    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:00.660846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.660809    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2ph\" (UniqueName: \"kubernetes.io/projected/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-kube-api-access-sl2ph\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv\" (UID: \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:00.661089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.661067    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv\" (UID: \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:00.723055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.723024    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:00.762609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.762544    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv\" (UID: \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:00.763444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.762818    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2ph\" (UniqueName: \"kubernetes.io/projected/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-kube-api-access-sl2ph\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv\" (UID: \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:00.763444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.763409    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv\" (UID: \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:00.776238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.776211    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2ph\" (UniqueName: \"kubernetes.io/projected/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-kube-api-access-sl2ph\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv\" (UID: \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:00.825346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.821792    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:00.826292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.825801    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:00.836809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.836589    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:00.845057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.844690    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:00.967362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.967334    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 11:00:00.967857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:00.967390    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 11:00:01.085369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.085301    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"]
Nov 03 11:00:01.194608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.186733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:01.238205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.236023    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"]
Nov 03 11:00:01.242061 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:01.241808    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04f7770_fc4a_4c82_a644_42f81a284965.slice/crio-e8d11226423d4d35ea5862d9f14a4223e2423e3549cfbbcb6c89df5ac8fb8831 WatchSource:0}: Error finding container e8d11226423d4d35ea5862d9f14a4223e2423e3549cfbbcb6c89df5ac8fb8831: Status 404 returned error can't find the container with id e8d11226423d4d35ea5862d9f14a4223e2423e3549cfbbcb6c89df5ac8fb8831
Nov 03 11:00:01.243147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.243073    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"]
Nov 03 11:00:01.250479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.250452    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"]
Nov 03 11:00:01.253651 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:01.253624    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4235c3_e211_4dae_93e7_d43181533579.slice/crio-ab42bb8b1fbc8f01fa9c9d8c717202441092cff96a8d22284961de62123d8558 WatchSource:0}: Error finding container ab42bb8b1fbc8f01fa9c9d8c717202441092cff96a8d22284961de62123d8558: Status 404 returned error can't find the container with id ab42bb8b1fbc8f01fa9c9d8c717202441092cff96a8d22284961de62123d8558
Nov 03 11:00:01.432163 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.432081    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"]
Nov 03 11:00:01.442140 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:01.442093    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e68bec_133c_441a_a470_fb99744a0bce.slice/crio-ac3512f00ede223181e6459144b0d9d75cd97d4f987c3b113a9f2c3ea1dd34c5 WatchSource:0}: Error finding container ac3512f00ede223181e6459144b0d9d75cd97d4f987c3b113a9f2c3ea1dd34c5: Status 404 returned error can't find the container with id ac3512f00ede223181e6459144b0d9d75cd97d4f987c3b113a9f2c3ea1dd34c5
Nov 03 11:00:01.480643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.480617    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"]
Nov 03 11:00:01.538498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.538472    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"]
Nov 03 11:00:01.624419 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:01.621823    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e9598e_3590_4926_b76d_e7e4b83fcf8a.slice/crio-2993c544c5cd44dd36c4f77b72ee6c7e2e84734889994afe0cc75e514490eaae WatchSource:0}: Error finding container 2993c544c5cd44dd36c4f77b72ee6c7e2e84734889994afe0cc75e514490eaae: Status 404 returned error can't find the container with id 2993c544c5cd44dd36c4f77b72ee6c7e2e84734889994afe0cc75e514490eaae
Nov 03 11:00:01.626435 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:01.626412    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9ccf95_7c78_4f2a_8e16_30745b8314c8.slice/crio-c5f503efb2dac0dd33706e7fe5053435e919d0a87b02e3fdf6e1ecca9939ef18 WatchSource:0}: Error finding container c5f503efb2dac0dd33706e7fe5053435e919d0a87b02e3fdf6e1ecca9939ef18: Status 404 returned error can't find the container with id c5f503efb2dac0dd33706e7fe5053435e919d0a87b02e3fdf6e1ecca9939ef18
Nov 03 11:00:01.626882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.626851    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"]
Nov 03 11:00:01.803658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.803622    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:01.966596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.966543    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 11:00:01.966786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:01.966646    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 11:00:02.101100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.101017    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" event={"ID":"b0e68bec-133c-441a-a470-fb99744a0bce","Type":"ContainerStarted","Data":"ac3512f00ede223181e6459144b0d9d75cd97d4f987c3b113a9f2c3ea1dd34c5"}
Nov 03 11:00:02.106717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.105650    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" event={"ID":"b858d6dd-e7ac-4895-84e2-fea16754cf28","Type":"ContainerStarted","Data":"a9f925b0046adf262d229a1af4f1257ba36e80de6bb5aa54d309e97cb70bd77f"}
Nov 03 11:00:02.106717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.105707    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" event={"ID":"b858d6dd-e7ac-4895-84e2-fea16754cf28","Type":"ContainerStarted","Data":"1b561010180b748679be8c8816e2bad885d859cd9cdac1ca2567343d70813c8f"}
Nov 03 11:00:02.109231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.109142    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" event={"ID":"7e9ccf95-7c78-4f2a-8e16-30745b8314c8","Type":"ContainerStarted","Data":"c5f503efb2dac0dd33706e7fe5053435e919d0a87b02e3fdf6e1ecca9939ef18"}
Nov 03 11:00:02.116827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.116801    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" event={"ID":"0a4235c3-e211-4dae-93e7-d43181533579","Type":"ContainerStarted","Data":"cfa6f306d68246c5e76d700cc507f841ac7c5f92a0a49ef424022247531cc399"}
Nov 03 11:00:02.116985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.116967    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" event={"ID":"0a4235c3-e211-4dae-93e7-d43181533579","Type":"ContainerStarted","Data":"ab42bb8b1fbc8f01fa9c9d8c717202441092cff96a8d22284961de62123d8558"}
Nov 03 11:00:02.120718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.120695    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" event={"ID":"025c2900-5963-4755-9c63-6dba56744a43","Type":"ContainerStarted","Data":"6462eaf973f22cce8459deda31a8b0a57ea2a40c80d18ef16eb8c207af494770"}
Nov 03 11:00:02.132927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.132788    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" event={"ID":"b0a9bb76-42d9-444e-997b-69edb5081787","Type":"ContainerStarted","Data":"66ca822f8699f1472acf7c1b59bdeedf3a582bd96b86884d1752360f1f9f4598"}
Nov 03 11:00:02.132927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.132839    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" event={"ID":"b0a9bb76-42d9-444e-997b-69edb5081787","Type":"ContainerStarted","Data":"d92965e39b4ddf3b754f68a1c776b394f127b41b1428f30221436642ed4005f6"}
Nov 03 11:00:02.133122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.133049    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" podStartSLOduration=2.133035769 podStartE2EDuration="2.133035769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.123206812 +0000 UTC m=+1832.328788865" watchObservedRunningTime="2026-11-03 11:00:02.133035769 +0000 UTC m=+1832.338617863"
Nov 03 11:00:02.135424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.135350    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" event={"ID":"21e9598e-3590-4926-b76d-e7e4b83fcf8a","Type":"ContainerStarted","Data":"2993c544c5cd44dd36c4f77b72ee6c7e2e84734889994afe0cc75e514490eaae"}
Nov 03 11:00:02.139433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.137852    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" event={"ID":"f04f7770-fc4a-4c82-a644-42f81a284965","Type":"ContainerStarted","Data":"08a33aa72171ecf45f6275f483079b7d5ddc6ef0d528a7cb2ecc68f092142060"}
Nov 03 11:00:02.139433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.137899    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" event={"ID":"f04f7770-fc4a-4c82-a644-42f81a284965","Type":"ContainerStarted","Data":"e8d11226423d4d35ea5862d9f14a4223e2423e3549cfbbcb6c89df5ac8fb8831"}
Nov 03 11:00:02.143654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.142149    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" podStartSLOduration=2.142137739 podStartE2EDuration="2.142137739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.138837327 +0000 UTC m=+1832.344419379" watchObservedRunningTime="2026-11-03 11:00:02.142137739 +0000 UTC m=+1832.347719790"
Nov 03 11:00:02.157938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.157903    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" podStartSLOduration=2.157891962 podStartE2EDuration="2.157891962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.154648605 +0000 UTC m=+1832.360230687" watchObservedRunningTime="2026-11-03 11:00:02.157891962 +0000 UTC m=+1832.363474012"
Nov 03 11:00:02.178099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.178053    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" podStartSLOduration=2.178039521 podStartE2EDuration="2.178039521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.172774177 +0000 UTC m=+1832.378356241" watchObservedRunningTime="2026-11-03 11:00:02.178039521 +0000 UTC m=+1832.383621572"
Nov 03 11:00:02.479950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.479912    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:02.648843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.648781    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:02.966819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.966789    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 11:00:02.966985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:02.966838    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 11:00:03.154536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.154503    5817 generic.go:356] "Generic (PLEG): container finished" podID="025c2900-5963-4755-9c63-6dba56744a43" containerID="ae2bd565e3bf1bec1012af59acda06c558ad72658e3f7bbbde3860195f5343cc" exitCode=60
Nov 03 11:00:03.155017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.154604    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" event={"ID":"025c2900-5963-4755-9c63-6dba56744a43","Type":"ContainerDied","Data":"ae2bd565e3bf1bec1012af59acda06c558ad72658e3f7bbbde3860195f5343cc"}
Nov 03 11:00:03.155017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.154970    5817 scope.go:122] "RemoveContainer" containerID="ae2bd565e3bf1bec1012af59acda06c558ad72658e3f7bbbde3860195f5343cc"
Nov 03 11:00:03.156925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.156863    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0a9bb76-42d9-444e-997b-69edb5081787" containerID="66ca822f8699f1472acf7c1b59bdeedf3a582bd96b86884d1752360f1f9f4598" exitCode=60
Nov 03 11:00:03.157018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.156996    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" event={"ID":"b0a9bb76-42d9-444e-997b-69edb5081787","Type":"ContainerDied","Data":"66ca822f8699f1472acf7c1b59bdeedf3a582bd96b86884d1752360f1f9f4598"}
Nov 03 11:00:03.157217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.157203    5817 scope.go:122] "RemoveContainer" containerID="66ca822f8699f1472acf7c1b59bdeedf3a582bd96b86884d1752360f1f9f4598"
Nov 03 11:00:03.159006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.158988    5817 generic.go:356] "Generic (PLEG): container finished" podID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerID="c4f2662fdc27e719058504676b5eb14807bb5e92187404cf5202cabd347b6a83" exitCode=60
Nov 03 11:00:03.159187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.159045    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" event={"ID":"21e9598e-3590-4926-b76d-e7e4b83fcf8a","Type":"ContainerDied","Data":"c4f2662fdc27e719058504676b5eb14807bb5e92187404cf5202cabd347b6a83"}
Nov 03 11:00:03.159304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.159269    5817 scope.go:122] "RemoveContainer" containerID="c4f2662fdc27e719058504676b5eb14807bb5e92187404cf5202cabd347b6a83"
Nov 03 11:00:03.171347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.171276    5817 generic.go:356] "Generic (PLEG): container finished" podID="f04f7770-fc4a-4c82-a644-42f81a284965" containerID="08a33aa72171ecf45f6275f483079b7d5ddc6ef0d528a7cb2ecc68f092142060" exitCode=60
Nov 03 11:00:03.172056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.171587    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" event={"ID":"f04f7770-fc4a-4c82-a644-42f81a284965","Type":"ContainerDied","Data":"08a33aa72171ecf45f6275f483079b7d5ddc6ef0d528a7cb2ecc68f092142060"}
Nov 03 11:00:03.172056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.171729    5817 scope.go:122] "RemoveContainer" containerID="08a33aa72171ecf45f6275f483079b7d5ddc6ef0d528a7cb2ecc68f092142060"
Nov 03 11:00:03.178186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.178163    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0e68bec-133c-441a-a470-fb99744a0bce" containerID="cd281ed335674347a3537260bbb648d75ca9d79d68519cb26b9d39061e02352c" exitCode=60
Nov 03 11:00:03.178314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.178254    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" event={"ID":"b0e68bec-133c-441a-a470-fb99744a0bce","Type":"ContainerDied","Data":"cd281ed335674347a3537260bbb648d75ca9d79d68519cb26b9d39061e02352c"}
Nov 03 11:00:03.178514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.178495    5817 scope.go:122] "RemoveContainer" containerID="cd281ed335674347a3537260bbb648d75ca9d79d68519cb26b9d39061e02352c"
Nov 03 11:00:03.184565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.184543    5817 generic.go:356] "Generic (PLEG): container finished" podID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerID="a9f925b0046adf262d229a1af4f1257ba36e80de6bb5aa54d309e97cb70bd77f" exitCode=60
Nov 03 11:00:03.185142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.184597    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" event={"ID":"b858d6dd-e7ac-4895-84e2-fea16754cf28","Type":"ContainerDied","Data":"a9f925b0046adf262d229a1af4f1257ba36e80de6bb5aa54d309e97cb70bd77f"}
Nov 03 11:00:03.185142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.184850    5817 scope.go:122] "RemoveContainer" containerID="a9f925b0046adf262d229a1af4f1257ba36e80de6bb5aa54d309e97cb70bd77f"
Nov 03 11:00:03.187195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.187175    5817 generic.go:356] "Generic (PLEG): container finished" podID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerID="ed9b7e61a9d0bb07761d021ae0768c8447c90ccf0d83234dac664c79c68e159f" exitCode=60
Nov 03 11:00:03.187311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.187250    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" event={"ID":"7e9ccf95-7c78-4f2a-8e16-30745b8314c8","Type":"ContainerDied","Data":"ed9b7e61a9d0bb07761d021ae0768c8447c90ccf0d83234dac664c79c68e159f"}
Nov 03 11:00:03.187755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.187453    5817 scope.go:122] "RemoveContainer" containerID="ed9b7e61a9d0bb07761d021ae0768c8447c90ccf0d83234dac664c79c68e159f"
Nov 03 11:00:03.189399 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.189381    5817 generic.go:356] "Generic (PLEG): container finished" podID="0a4235c3-e211-4dae-93e7-d43181533579" containerID="cfa6f306d68246c5e76d700cc507f841ac7c5f92a0a49ef424022247531cc399" exitCode=60
Nov 03 11:00:03.189505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.189409    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" event={"ID":"0a4235c3-e211-4dae-93e7-d43181533579","Type":"ContainerDied","Data":"cfa6f306d68246c5e76d700cc507f841ac7c5f92a0a49ef424022247531cc399"}
Nov 03 11:00:03.189750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.189734    5817 scope.go:122] "RemoveContainer" containerID="cfa6f306d68246c5e76d700cc507f841ac7c5f92a0a49ef424022247531cc399"
Nov 03 11:00:03.296451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.296385    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:03.958883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:03.958849    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:04.009978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.008866    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 11:00:04.009978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.009940    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 11:00:04.013040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.012937    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 11:00:04.216138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.216102    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" event={"ID":"025c2900-5963-4755-9c63-6dba56744a43","Type":"ContainerStarted","Data":"22748731ca89389fbc9f6a90a3452648bc4376922b29e4b9d79e12ec6885183b"}
Nov 03 11:00:04.219008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.218978    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" event={"ID":"b0a9bb76-42d9-444e-997b-69edb5081787","Type":"ContainerStarted","Data":"6c50545b2e204d25b2f7e818f75246871c5905fcb5e42c1849feb525060706a6"}
Nov 03 11:00:04.221649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.221622    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" event={"ID":"21e9598e-3590-4926-b76d-e7e4b83fcf8a","Type":"ContainerStarted","Data":"1f0904dea169fd5dc865e3757a05bc836f20dbe2a7a392ee5fa0bbf38d531456"}
Nov 03 11:00:04.224493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.224455    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" event={"ID":"f04f7770-fc4a-4c82-a644-42f81a284965","Type":"ContainerStarted","Data":"4485475eff14df2e404b0fffbf6c3556e6e6760ae362f71d0fa72bf903bf04e2"}
Nov 03 11:00:04.226739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.226719    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" event={"ID":"b0e68bec-133c-441a-a470-fb99744a0bce","Type":"ContainerStarted","Data":"278dcd2009e50406784cb7b9dbd5a85b7aca3b93ee1d15d80ffd3adbcf18eb5d"}
Nov 03 11:00:04.229190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.229163    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" event={"ID":"b858d6dd-e7ac-4895-84e2-fea16754cf28","Type":"ContainerStarted","Data":"68647287894b8fa8610bd1a617ef132ef240f1a1ba6b41ffa33772da621b5bd2"}
Nov 03 11:00:04.231506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.231483    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" event={"ID":"7e9ccf95-7c78-4f2a-8e16-30745b8314c8","Type":"ContainerStarted","Data":"82863d90262a5d4e20af51a894655f5513e2a13f07455b993da129900da48e3c"}
Nov 03 11:00:04.234166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.234129    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" event={"ID":"0a4235c3-e211-4dae-93e7-d43181533579","Type":"ContainerStarted","Data":"a51bddc4290d773e5f60b31b2192bf9f6d8b6d7c61993e59e0d0e59921403171"}
Nov 03 11:00:04.239481 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.239442    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" podStartSLOduration=4.239430488 podStartE2EDuration="4.239430488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:04.236103883 +0000 UTC m=+1834.441685935" watchObservedRunningTime="2026-11-03 11:00:04.239430488 +0000 UTC m=+1834.445012539"
Nov 03 11:00:04.309394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.309331    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" podStartSLOduration=4.309313508 podStartE2EDuration="4.309313508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:04.306545183 +0000 UTC m=+1834.512127235" watchObservedRunningTime="2026-11-03 11:00:04.309313508 +0000 UTC m=+1834.514895559"
Nov 03 11:00:04.342244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.342197    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" podStartSLOduration=4.342184029 podStartE2EDuration="4.342184029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:04.339377859 +0000 UTC m=+1834.544959911" watchObservedRunningTime="2026-11-03 11:00:04.342184029 +0000 UTC m=+1834.547766077"
Nov 03 11:00:04.354916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.354880    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" podStartSLOduration=4.354868598 podStartE2EDuration="4.354868598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:04.354257887 +0000 UTC m=+1834.559839937" watchObservedRunningTime="2026-11-03 11:00:04.354868598 +0000 UTC m=+1834.560450686"
Nov 03 11:00:04.751396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:04.751221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:05.247237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.247202    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0e68bec-133c-441a-a470-fb99744a0bce" containerID="278dcd2009e50406784cb7b9dbd5a85b7aca3b93ee1d15d80ffd3adbcf18eb5d" exitCode=60
Nov 03 11:00:05.247765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.247290    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" event={"ID":"b0e68bec-133c-441a-a470-fb99744a0bce","Type":"ContainerDied","Data":"278dcd2009e50406784cb7b9dbd5a85b7aca3b93ee1d15d80ffd3adbcf18eb5d"}
Nov 03 11:00:05.247765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.247355    5817 scope.go:122] "RemoveContainer" containerID="cd281ed335674347a3537260bbb648d75ca9d79d68519cb26b9d39061e02352c"
Nov 03 11:00:05.247765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.247552    5817 scope.go:122] "RemoveContainer" containerID="278dcd2009e50406784cb7b9dbd5a85b7aca3b93ee1d15d80ffd3adbcf18eb5d"
Nov 03 11:00:05.247941 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:05.247913    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9_odh-ai-gateway-infra(b0e68bec-133c-441a-a470-fb99744a0bce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" podUID="b0e68bec-133c-441a-a470-fb99744a0bce"
Nov 03 11:00:05.249767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.249740    5817 generic.go:356] "Generic (PLEG): container finished" podID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerID="68647287894b8fa8610bd1a617ef132ef240f1a1ba6b41ffa33772da621b5bd2" exitCode=60
Nov 03 11:00:05.249896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.249824    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" event={"ID":"b858d6dd-e7ac-4895-84e2-fea16754cf28","Type":"ContainerDied","Data":"68647287894b8fa8610bd1a617ef132ef240f1a1ba6b41ffa33772da621b5bd2"}
Nov 03 11:00:05.250089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.250070    5817 scope.go:122] "RemoveContainer" containerID="68647287894b8fa8610bd1a617ef132ef240f1a1ba6b41ffa33772da621b5bd2"
Nov 03 11:00:05.250444 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:05.250384    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7_odh-ai-gateway-infra(b858d6dd-e7ac-4895-84e2-fea16754cf28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28"
Nov 03 11:00:05.251871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.251836    5817 generic.go:356] "Generic (PLEG): container finished" podID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerID="82863d90262a5d4e20af51a894655f5513e2a13f07455b993da129900da48e3c" exitCode=60
Nov 03 11:00:05.251984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.251915    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" event={"ID":"7e9ccf95-7c78-4f2a-8e16-30745b8314c8","Type":"ContainerDied","Data":"82863d90262a5d4e20af51a894655f5513e2a13f07455b993da129900da48e3c"}
Nov 03 11:00:05.252196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.252174    5817 scope.go:122] "RemoveContainer" containerID="82863d90262a5d4e20af51a894655f5513e2a13f07455b993da129900da48e3c"
Nov 03 11:00:05.252613 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:05.252551    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv_odh-ai-gateway-infra(7e9ccf95-7c78-4f2a-8e16-30745b8314c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8"
Nov 03 11:00:05.254225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.254207    5817 generic.go:356] "Generic (PLEG): container finished" podID="0a4235c3-e211-4dae-93e7-d43181533579" containerID="a51bddc4290d773e5f60b31b2192bf9f6d8b6d7c61993e59e0d0e59921403171" exitCode=60
Nov 03 11:00:05.254336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.254293    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" event={"ID":"0a4235c3-e211-4dae-93e7-d43181533579","Type":"ContainerDied","Data":"a51bddc4290d773e5f60b31b2192bf9f6d8b6d7c61993e59e0d0e59921403171"}
Nov 03 11:00:05.254525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.254507    5817 scope.go:122] "RemoveContainer" containerID="a51bddc4290d773e5f60b31b2192bf9f6d8b6d7c61993e59e0d0e59921403171"
Nov 03 11:00:05.254794 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:05.254774    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9_odh-ai-gateway-infra(0a4235c3-e211-4dae-93e7-d43181533579)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" podUID="0a4235c3-e211-4dae-93e7-d43181533579"
Nov 03 11:00:05.256355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.256336    5817 generic.go:356] "Generic (PLEG): container finished" podID="025c2900-5963-4755-9c63-6dba56744a43" containerID="22748731ca89389fbc9f6a90a3452648bc4376922b29e4b9d79e12ec6885183b" exitCode=60
Nov 03 11:00:05.256444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.256413    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" event={"ID":"025c2900-5963-4755-9c63-6dba56744a43","Type":"ContainerDied","Data":"22748731ca89389fbc9f6a90a3452648bc4376922b29e4b9d79e12ec6885183b"}
Nov 03 11:00:05.256649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.256632    5817 scope.go:122] "RemoveContainer" containerID="22748731ca89389fbc9f6a90a3452648bc4376922b29e4b9d79e12ec6885183b"
Nov 03 11:00:05.256908 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:05.256880    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5_odh-ai-gateway-infra(025c2900-5963-4755-9c63-6dba56744a43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" podUID="025c2900-5963-4755-9c63-6dba56744a43"
Nov 03 11:00:05.258369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.258350    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0a9bb76-42d9-444e-997b-69edb5081787" containerID="6c50545b2e204d25b2f7e818f75246871c5905fcb5e42c1849feb525060706a6" exitCode=60
Nov 03 11:00:05.258458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.258419    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" event={"ID":"b0a9bb76-42d9-444e-997b-69edb5081787","Type":"ContainerDied","Data":"6c50545b2e204d25b2f7e818f75246871c5905fcb5e42c1849feb525060706a6"}
Nov 03 11:00:05.258660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.258635    5817 scope.go:122] "RemoveContainer" containerID="6c50545b2e204d25b2f7e818f75246871c5905fcb5e42c1849feb525060706a6"
Nov 03 11:00:05.258907 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:05.258879    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895060-8lw9p_odh-ai-gateway-infra(b0a9bb76-42d9-444e-997b-69edb5081787)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" podUID="b0a9bb76-42d9-444e-997b-69edb5081787"
Nov 03 11:00:05.260368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.260349    5817 generic.go:356] "Generic (PLEG): container finished" podID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerID="1f0904dea169fd5dc865e3757a05bc836f20dbe2a7a392ee5fa0bbf38d531456" exitCode=60
Nov 03 11:00:05.260470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.260375    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" event={"ID":"21e9598e-3590-4926-b76d-e7e4b83fcf8a","Type":"ContainerDied","Data":"1f0904dea169fd5dc865e3757a05bc836f20dbe2a7a392ee5fa0bbf38d531456"}
Nov 03 11:00:05.260704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.260687    5817 scope.go:122] "RemoveContainer" containerID="1f0904dea169fd5dc865e3757a05bc836f20dbe2a7a392ee5fa0bbf38d531456"
Nov 03 11:00:05.260955 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:05.260934    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr_odh-ai-gateway-infra(21e9598e-3590-4926-b76d-e7e4b83fcf8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a"
Nov 03 11:00:05.262518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.262493    5817 generic.go:356] "Generic (PLEG): container finished" podID="f04f7770-fc4a-4c82-a644-42f81a284965" containerID="4485475eff14df2e404b0fffbf6c3556e6e6760ae362f71d0fa72bf903bf04e2" exitCode=60
Nov 03 11:00:05.262977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.262949    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" event={"ID":"f04f7770-fc4a-4c82-a644-42f81a284965","Type":"ContainerDied","Data":"4485475eff14df2e404b0fffbf6c3556e6e6760ae362f71d0fa72bf903bf04e2"}
Nov 03 11:00:05.263266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.263250    5817 scope.go:122] "RemoveContainer" containerID="4485475eff14df2e404b0fffbf6c3556e6e6760ae362f71d0fa72bf903bf04e2"
Nov 03 11:00:05.263519 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:05.263500    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk_odh-ai-gateway-infra(f04f7770-fc4a-4c82-a644-42f81a284965)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" podUID="f04f7770-fc4a-4c82-a644-42f81a284965"
Nov 03 11:00:05.264323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.264307    5817 scope.go:122] "RemoveContainer" containerID="a9f925b0046adf262d229a1af4f1257ba36e80de6bb5aa54d309e97cb70bd77f"
Nov 03 11:00:05.280194 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.280172    5817 scope.go:122] "RemoveContainer" containerID="ed9b7e61a9d0bb07761d021ae0768c8447c90ccf0d83234dac664c79c68e159f"
Nov 03 11:00:05.297811 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.297784    5817 scope.go:122] "RemoveContainer" containerID="cfa6f306d68246c5e76d700cc507f841ac7c5f92a0a49ef424022247531cc399"
Nov 03 11:00:05.318779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.318757    5817 scope.go:122] "RemoveContainer" containerID="ae2bd565e3bf1bec1012af59acda06c558ad72658e3f7bbbde3860195f5343cc"
Nov 03 11:00:05.343443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.343421    5817 scope.go:122] "RemoveContainer" containerID="66ca822f8699f1472acf7c1b59bdeedf3a582bd96b86884d1752360f1f9f4598"
Nov 03 11:00:05.360357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.360339    5817 scope.go:122] "RemoveContainer" containerID="c4f2662fdc27e719058504676b5eb14807bb5e92187404cf5202cabd347b6a83"
Nov 03 11:00:05.375714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.375692    5817 scope.go:122] "RemoveContainer" containerID="08a33aa72171ecf45f6275f483079b7d5ddc6ef0d528a7cb2ecc68f092142060"
Nov 03 11:00:05.815379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:05.815336    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:06.274603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.274538    5817 scope.go:122] "RemoveContainer" containerID="278dcd2009e50406784cb7b9dbd5a85b7aca3b93ee1d15d80ffd3adbcf18eb5d"
Nov 03 11:00:06.275061 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:06.274902    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9_odh-ai-gateway-infra(b0e68bec-133c-441a-a470-fb99744a0bce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" podUID="b0e68bec-133c-441a-a470-fb99744a0bce"
Nov 03 11:00:06.276532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.276510    5817 scope.go:122] "RemoveContainer" containerID="68647287894b8fa8610bd1a617ef132ef240f1a1ba6b41ffa33772da621b5bd2"
Nov 03 11:00:06.276833 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:06.276811    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7_odh-ai-gateway-infra(b858d6dd-e7ac-4895-84e2-fea16754cf28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28"
Nov 03 11:00:06.278331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.278314    5817 scope.go:122] "RemoveContainer" containerID="82863d90262a5d4e20af51a894655f5513e2a13f07455b993da129900da48e3c"
Nov 03 11:00:06.278637 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:06.278615    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv_odh-ai-gateway-infra(7e9ccf95-7c78-4f2a-8e16-30745b8314c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8"
Nov 03 11:00:06.279937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.279920    5817 scope.go:122] "RemoveContainer" containerID="a51bddc4290d773e5f60b31b2192bf9f6d8b6d7c61993e59e0d0e59921403171"
Nov 03 11:00:06.280174 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:06.280152    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9_odh-ai-gateway-infra(0a4235c3-e211-4dae-93e7-d43181533579)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" podUID="0a4235c3-e211-4dae-93e7-d43181533579"
Nov 03 11:00:06.281490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.281472    5817 scope.go:122] "RemoveContainer" containerID="22748731ca89389fbc9f6a90a3452648bc4376922b29e4b9d79e12ec6885183b"
Nov 03 11:00:06.281704 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:06.281688    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5_odh-ai-gateway-infra(025c2900-5963-4755-9c63-6dba56744a43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" podUID="025c2900-5963-4755-9c63-6dba56744a43"
Nov 03 11:00:06.283256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.283238    5817 scope.go:122] "RemoveContainer" containerID="6c50545b2e204d25b2f7e818f75246871c5905fcb5e42c1849feb525060706a6"
Nov 03 11:00:06.283470 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:06.283447    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895060-8lw9p_odh-ai-gateway-infra(b0a9bb76-42d9-444e-997b-69edb5081787)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" podUID="b0a9bb76-42d9-444e-997b-69edb5081787"
Nov 03 11:00:06.284936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.284918    5817 scope.go:122] "RemoveContainer" containerID="1f0904dea169fd5dc865e3757a05bc836f20dbe2a7a392ee5fa0bbf38d531456"
Nov 03 11:00:06.285131 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:06.285113    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr_odh-ai-gateway-infra(21e9598e-3590-4926-b76d-e7e4b83fcf8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a"
Nov 03 11:00:06.286439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.286425    5817 scope.go:122] "RemoveContainer" containerID="4485475eff14df2e404b0fffbf6c3556e6e6760ae362f71d0fa72bf903bf04e2"
Nov 03 11:00:06.286655 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:06.286637    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk_odh-ai-gateway-infra(f04f7770-fc4a-4c82-a644-42f81a284965)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" podUID="f04f7770-fc4a-4c82-a644-42f81a284965"
Nov 03 11:00:06.523601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:06.523130    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:07.155992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:07.149222    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:07.755913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:07.755873    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:08.439407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:08.439368    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:08.574561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:08.574518    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:09.181278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:09.181235    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:09.875247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:09.875202    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:10.575631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:10.574174    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:11.282204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:11.278140    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:11.857597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:11.857275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:12.469921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:12.469883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:13.073694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:13.073563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:13.680330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:13.680294    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:14.349640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:14.349608    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:14.365261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:14.365229    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:00:14.895049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:14.895015    5817 scope.go:122] "RemoveContainer" containerID="1f0904dea169fd5dc865e3757a05bc836f20dbe2a7a392ee5fa0bbf38d531456"
Nov 03 11:00:14.895841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:14.895463    5817 scope.go:122] "RemoveContainer" containerID="278dcd2009e50406784cb7b9dbd5a85b7aca3b93ee1d15d80ffd3adbcf18eb5d"
Nov 03 11:00:15.144517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:15.144477    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:15.388319 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:15.388290    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" event={"ID":"b0e68bec-133c-441a-a470-fb99744a0bce","Type":"ContainerStarted","Data":"fe6f128f4cf1549d67cfa799cc2cb0d28832a98e5f431ca4fc092fa371f8eea4"}
Nov 03 11:00:15.906651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:15.903609    5817 scope.go:122] "RemoveContainer" containerID="22748731ca89389fbc9f6a90a3452648bc4376922b29e4b9d79e12ec6885183b"
Nov 03 11:00:15.928722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:15.927783    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:16.055091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.054848    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:16.256598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.252139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:16.473062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.473031    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"]
Nov 03 11:00:16.477606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.477544    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0e68bec-133c-441a-a470-fb99744a0bce" containerID="fe6f128f4cf1549d67cfa799cc2cb0d28832a98e5f431ca4fc092fa371f8eea4" exitCode=60
Nov 03 11:00:16.478897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.477633    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" event={"ID":"b0e68bec-133c-441a-a470-fb99744a0bce","Type":"ContainerDied","Data":"fe6f128f4cf1549d67cfa799cc2cb0d28832a98e5f431ca4fc092fa371f8eea4"}
Nov 03 11:00:16.479070 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.478979    5817 scope.go:122] "RemoveContainer" containerID="278dcd2009e50406784cb7b9dbd5a85b7aca3b93ee1d15d80ffd3adbcf18eb5d"
Nov 03 11:00:16.482019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.481994    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" event={"ID":"025c2900-5963-4755-9c63-6dba56744a43","Type":"ContainerStarted","Data":"d03ebe3d3ad0a2963b2d3d9a8599f75ac099a6632e1454e3dfacfcd87f1c547c"}
Nov 03 11:00:16.484116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.484090    5817 generic.go:356] "Generic (PLEG): container finished" podID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerID="dc16826f171722572104d08812c3b85be3eaa132bd7dbdc8199b5186239f3d22" exitCode=60
Nov 03 11:00:16.484274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.484246    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" event={"ID":"21e9598e-3590-4926-b76d-e7e4b83fcf8a","Type":"ContainerDied","Data":"dc16826f171722572104d08812c3b85be3eaa132bd7dbdc8199b5186239f3d22"}
Nov 03 11:00:16.484971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.484956    5817 scope.go:122] "RemoveContainer" containerID="dc16826f171722572104d08812c3b85be3eaa132bd7dbdc8199b5186239f3d22"
Nov 03 11:00:16.485450 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:16.485422    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr_odh-ai-gateway-infra(21e9598e-3590-4926-b76d-e7e4b83fcf8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a"
Nov 03 11:00:16.529992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.529964    5817 scope.go:122] "RemoveContainer" containerID="1f0904dea169fd5dc865e3757a05bc836f20dbe2a7a392ee5fa0bbf38d531456"
Nov 03 11:00:16.783811 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.783784    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:16.885827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.885789    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0e68bec-133c-441a-a470-fb99744a0bce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e68bec-133c-441a-a470-fb99744a0bce-maas-api-service-ca\") pod \"b0e68bec-133c-441a-a470-fb99744a0bce\" (UID: \"b0e68bec-133c-441a-a470-fb99744a0bce\") "
Nov 03 11:00:16.886093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.886072    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e68bec-133c-441a-a470-fb99744a0bce-kube-api-access-47qgp\" (UniqueName: \"kubernetes.io/projected/b0e68bec-133c-441a-a470-fb99744a0bce-kube-api-access-47qgp\") pod \"b0e68bec-133c-441a-a470-fb99744a0bce\" (UID: \"b0e68bec-133c-441a-a470-fb99744a0bce\") "
Nov 03 11:00:16.886226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.886125    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e68bec-133c-441a-a470-fb99744a0bce-maas-api-service-ca" pod "b0e68bec-133c-441a-a470-fb99744a0bce" (UID: "b0e68bec-133c-441a-a470-fb99744a0bce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:16.897602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.886747    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e68bec-133c-441a-a470-fb99744a0bce-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:16.897602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.895031    5817 scope.go:122] "RemoveContainer" containerID="68647287894b8fa8610bd1a617ef132ef240f1a1ba6b41ffa33772da621b5bd2"
Nov 03 11:00:16.898024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.897997    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e68bec-133c-441a-a470-fb99744a0bce-kube-api-access-47qgp" pod "b0e68bec-133c-441a-a470-fb99744a0bce" (UID: "b0e68bec-133c-441a-a470-fb99744a0bce"). InnerVolumeSpecName "kube-api-access-47qgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:16.986724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.986509    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:16.991791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:16.988283    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qgp\" (UniqueName: \"kubernetes.io/projected/b0e68bec-133c-441a-a470-fb99744a0bce-kube-api-access-47qgp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:17.322132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.322040    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"]
Nov 03 11:00:17.323843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.323795    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.324069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.324054    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.324232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.324219    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.324350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.324338    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.324476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.324465    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.324585 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.324561    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.324913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.324883    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.325064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.325052    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.325196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.325184    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" containerName="cleanup"
Nov 03 11:00:17.330513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.330495    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.330807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.330784    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"]
Nov 03 11:00:17.334475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.334452    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:00:17.404546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.402564    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.404546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.402760    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.404546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.402912    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6gl\" (UniqueName: \"kubernetes.io/projected/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-kube-api-access-bf6gl\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.511101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.510562    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.511846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.511804    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6gl\" (UniqueName: \"kubernetes.io/projected/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-kube-api-access-bf6gl\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.512131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.512108    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.513534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.513115    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.532767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.532739    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"
Nov 03 11:00:17.532892 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.532751    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9" event={"ID":"b0e68bec-133c-441a-a470-fb99744a0bce","Type":"ContainerDied","Data":"ac3512f00ede223181e6459144b0d9d75cd97d4f987c3b113a9f2c3ea1dd34c5"}
Nov 03 11:00:17.532892 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.532861    5817 scope.go:122] "RemoveContainer" containerID="fe6f128f4cf1549d67cfa799cc2cb0d28832a98e5f431ca4fc092fa371f8eea4"
Nov 03 11:00:17.536064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.536023    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6gl\" (UniqueName: \"kubernetes.io/projected/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-kube-api-access-bf6gl\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.537433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.536777    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" event={"ID":"b858d6dd-e7ac-4895-84e2-fea16754cf28","Type":"ContainerStarted","Data":"f29359e4551ae3ae01e119e9dbb4079133a4fc4e22e64915af91838db0dd9de0"}
Nov 03 11:00:17.542620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.540264    5817 generic.go:356] "Generic (PLEG): container finished" podID="025c2900-5963-4755-9c63-6dba56744a43" containerID="d03ebe3d3ad0a2963b2d3d9a8599f75ac099a6632e1454e3dfacfcd87f1c547c" exitCode=60
Nov 03 11:00:17.542620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.540387    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" event={"ID":"025c2900-5963-4755-9c63-6dba56744a43","Type":"ContainerDied","Data":"d03ebe3d3ad0a2963b2d3d9a8599f75ac099a6632e1454e3dfacfcd87f1c547c"}
Nov 03 11:00:17.542620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.540791    5817 scope.go:122] "RemoveContainer" containerID="d03ebe3d3ad0a2963b2d3d9a8599f75ac099a6632e1454e3dfacfcd87f1c547c"
Nov 03 11:00:17.542620 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:17.541083    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5_odh-ai-gateway-infra(025c2900-5963-4755-9c63-6dba56744a43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" podUID="025c2900-5963-4755-9c63-6dba56744a43"
Nov 03 11:00:17.550927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.550902    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7e50bf-4srbb\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.554891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.554865    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"]
Nov 03 11:00:17.571502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.571474    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"]
Nov 03 11:00:17.645739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.645706    5817 scope.go:122] "RemoveContainer" containerID="22748731ca89389fbc9f6a90a3452648bc4376922b29e4b9d79e12ec6885183b"
Nov 03 11:00:17.655136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.655109    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:17.683524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.683496    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"]
Nov 03 11:00:17.721616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.720037    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895060-km5m9"]
Nov 03 11:00:17.957600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.937256    5817 scope.go:122] "RemoveContainer" containerID="82863d90262a5d4e20af51a894655f5513e2a13f07455b993da129900da48e3c"
Nov 03 11:00:17.957600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.938366    5817 scope.go:122] "RemoveContainer" containerID="a51bddc4290d773e5f60b31b2192bf9f6d8b6d7c61993e59e0d0e59921403171"
Nov 03 11:00:17.957600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.939180    5817 scope.go:122] "RemoveContainer" containerID="6c50545b2e204d25b2f7e818f75246871c5905fcb5e42c1849feb525060706a6"
Nov 03 11:00:17.978998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:17.978887    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:18.050771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.050724    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0e68bec-133c-441a-a470-fb99744a0bce" path="/var/lib/kubelet/pods/b0e68bec-133c-441a-a470-fb99744a0bce/volumes"
Nov 03 11:00:18.051466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.051438    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"]
Nov 03 11:00:18.051589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.051482    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad0d35-cvm29"]
Nov 03 11:00:18.051963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.051943    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:18.145397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.144327    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e9598e-3590-4926-b76d-e7e4b83fcf8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e9598e-3590-4926-b76d-e7e4b83fcf8a-maas-api-service-ca\") pod \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\" (UID: \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\") "
Nov 03 11:00:18.145397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.144423    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e9598e-3590-4926-b76d-e7e4b83fcf8a-kube-api-access-sp9sk\" (UniqueName: \"kubernetes.io/projected/21e9598e-3590-4926-b76d-e7e4b83fcf8a-kube-api-access-sp9sk\") pod \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\" (UID: \"21e9598e-3590-4926-b76d-e7e4b83fcf8a\") "
Nov 03 11:00:18.145397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.144796    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e9598e-3590-4926-b76d-e7e4b83fcf8a-maas-api-service-ca" pod "21e9598e-3590-4926-b76d-e7e4b83fcf8a" (UID: "21e9598e-3590-4926-b76d-e7e4b83fcf8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:18.145397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.145360    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e9598e-3590-4926-b76d-e7e4b83fcf8a-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:18.147894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.147844    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e9598e-3590-4926-b76d-e7e4b83fcf8a-kube-api-access-sp9sk" pod "21e9598e-3590-4926-b76d-e7e4b83fcf8a" (UID: "21e9598e-3590-4926-b76d-e7e4b83fcf8a"). InnerVolumeSpecName "kube-api-access-sp9sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:18.246866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.246812    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp9sk\" (UniqueName: \"kubernetes.io/projected/21e9598e-3590-4926-b76d-e7e4b83fcf8a-kube-api-access-sp9sk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:18.258689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.258662    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"]
Nov 03 11:00:18.574176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.574139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:18.581604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.581140    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb" event={"ID":"5a24b2a0-d4ad-41dc-981a-5c735541cfa3","Type":"ContainerStarted","Data":"09398a8e538ba394eabe5d872985205cce00051a553c0b67e73d0fe814130bad"}
Nov 03 11:00:18.600617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.600216    5817 generic.go:356] "Generic (PLEG): container finished" podID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerID="f29359e4551ae3ae01e119e9dbb4079133a4fc4e22e64915af91838db0dd9de0" exitCode=60
Nov 03 11:00:18.600617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.600283    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" event={"ID":"b858d6dd-e7ac-4895-84e2-fea16754cf28","Type":"ContainerDied","Data":"f29359e4551ae3ae01e119e9dbb4079133a4fc4e22e64915af91838db0dd9de0"}
Nov 03 11:00:18.600617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.600371    5817 scope.go:122] "RemoveContainer" containerID="68647287894b8fa8610bd1a617ef132ef240f1a1ba6b41ffa33772da621b5bd2"
Nov 03 11:00:18.601363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.601092    5817 scope.go:122] "RemoveContainer" containerID="f29359e4551ae3ae01e119e9dbb4079133a4fc4e22e64915af91838db0dd9de0"
Nov 03 11:00:18.614348 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:18.614273    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7_odh-ai-gateway-infra(b858d6dd-e7ac-4895-84e2-fea16754cf28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28"
Nov 03 11:00:18.623049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.622972    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"]
Nov 03 11:00:18.660030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.660002    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr" event={"ID":"21e9598e-3590-4926-b76d-e7e4b83fcf8a","Type":"ContainerDied","Data":"2993c544c5cd44dd36c4f77b72ee6c7e2e84734889994afe0cc75e514490eaae"}
Nov 03 11:00:18.660370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.660172    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"
Nov 03 11:00:18.670956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.669779    5817 scope.go:122] "RemoveContainer" containerID="dc16826f171722572104d08812c3b85be3eaa132bd7dbdc8199b5186239f3d22"
Nov 03 11:00:18.755154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.755125    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"]
Nov 03 11:00:18.759791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.759768    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895060-nw8lr"]
Nov 03 11:00:18.920610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:18.901987    5817 scope.go:122] "RemoveContainer" containerID="4485475eff14df2e404b0fffbf6c3556e6e6760ae362f71d0fa72bf903bf04e2"
Nov 03 11:00:19.028658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.027867    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:19.181545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.181516    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025c2900-5963-4755-9c63-6dba56744a43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025c2900-5963-4755-9c63-6dba56744a43-maas-api-service-ca\") pod \"025c2900-5963-4755-9c63-6dba56744a43\" (UID: \"025c2900-5963-4755-9c63-6dba56744a43\") "
Nov 03 11:00:19.181740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.181613    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025c2900-5963-4755-9c63-6dba56744a43-kube-api-access-gtjh5\" (UniqueName: \"kubernetes.io/projected/025c2900-5963-4755-9c63-6dba56744a43-kube-api-access-gtjh5\") pod \"025c2900-5963-4755-9c63-6dba56744a43\" (UID: \"025c2900-5963-4755-9c63-6dba56744a43\") "
Nov 03 11:00:19.181967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.181940    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025c2900-5963-4755-9c63-6dba56744a43-maas-api-service-ca" pod "025c2900-5963-4755-9c63-6dba56744a43" (UID: "025c2900-5963-4755-9c63-6dba56744a43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:19.182345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.182325    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025c2900-5963-4755-9c63-6dba56744a43-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.183933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.183909    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025c2900-5963-4755-9c63-6dba56744a43-kube-api-access-gtjh5" pod "025c2900-5963-4755-9c63-6dba56744a43" (UID: "025c2900-5963-4755-9c63-6dba56744a43"). InnerVolumeSpecName "kube-api-access-gtjh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:19.220334 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.220299    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:19.283538 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.283507    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtjh5\" (UniqueName: \"kubernetes.io/projected/025c2900-5963-4755-9c63-6dba56744a43-kube-api-access-gtjh5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.354606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.351452    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:19.583489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.583451    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"]
Nov 03 11:00:19.584451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584427    5817 cpu_manager.go:414] "containerMap: removing container" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:19.584451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584451    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:19.584649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584484    5817 cpu_manager.go:414] "containerMap: removing container" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:19.584649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584492    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:19.584649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584523    5817 cpu_manager.go:414] "containerMap: removing container" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:19.584649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584532    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:19.584649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584617    5817 cpu_manager.go:414] "containerMap: removing container" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:19.584649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584626    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:19.584943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584803    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:19.584943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584821    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:19.584943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584836    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:19.584943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.584854    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:19.654010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.653980    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:19.655248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.655222    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"]
Nov 03 11:00:19.659359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.659322    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-185908-openshift-default-dockercfg-88npj\""
Nov 03 11:00:19.942598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.939429    5817 generic.go:356] "Generic (PLEG): container finished" podID="5a24b2a0-d4ad-41dc-981a-5c735541cfa3" containerID="246d24a4c4569035cb332e684c73d28de56a73cf97f1be6d56a59d2ca9a3e263" exitCode=0
Nov 03 11:00:19.968597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:19.943146    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb" event={"ID":"5a24b2a0-d4ad-41dc-981a-5c735541cfa3","Type":"ContainerDied","Data":"246d24a4c4569035cb332e684c73d28de56a73cf97f1be6d56a59d2ca9a3e263"}
Nov 03 11:00:20.043870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.041536    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.144687    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-envoy\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.144883    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-podinfo\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.144926    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-data\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.144985    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-socket\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.145014    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-certs\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.145082    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033ade0f-f138-4a4b-b2b8-41d02673adab-istiod-ca-cert\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.145118    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-credential-socket\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.145215    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-token\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.145599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.145248    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4nvf\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-kube-api-access-q4nvf\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.163866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.163832    5817 generic.go:356] "Generic (PLEG): container finished" podID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerID="dc0df8101da7da0715f7fba8ce80a63cb05dd480c6b6f06b81f6926c3dcf038a" exitCode=60
Nov 03 11:00:20.170831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.169153    5817 generic.go:356] "Generic (PLEG): container finished" podID="0a4235c3-e211-4dae-93e7-d43181533579" containerID="d85fd83e6b2c86c435d938a852d3972b9f0e05f4df807e2ebf112747a61fa1a9" exitCode=60
Nov 03 11:00:20.175993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.175921    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"
Nov 03 11:00:20.176448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.176385    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15465683-12e9-4bb3-9592-5e5abe524338" path="/var/lib/kubelet/pods/15465683-12e9-4bb3-9592-5e5abe524338/volumes"
Nov 03 11:00:20.198594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.177158    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" path="/var/lib/kubelet/pods/21e9598e-3590-4926-b76d-e7e4b83fcf8a/volumes"
Nov 03 11:00:20.199783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.199754    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" event={"ID":"7e9ccf95-7c78-4f2a-8e16-30745b8314c8","Type":"ContainerDied","Data":"dc0df8101da7da0715f7fba8ce80a63cb05dd480c6b6f06b81f6926c3dcf038a"}
Nov 03 11:00:20.199997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.199972    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" event={"ID":"0a4235c3-e211-4dae-93e7-d43181533579","Type":"ContainerDied","Data":"d85fd83e6b2c86c435d938a852d3972b9f0e05f4df807e2ebf112747a61fa1a9"}
Nov 03 11:00:20.200134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.200117    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5" event={"ID":"025c2900-5963-4755-9c63-6dba56744a43","Type":"ContainerDied","Data":"6462eaf973f22cce8459deda31a8b0a57ea2a40c80d18ef16eb8c207af494770"}
Nov 03 11:00:20.200256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.200240    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" event={"ID":"b0a9bb76-42d9-444e-997b-69edb5081787","Type":"ContainerDied","Data":"c25992503e6b6ff4dd8cdb9d0123835759ee5d7e4a3778fed6fb10015ff84db6"}
Nov 03 11:00:20.202675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.202656    5817 scope.go:122] "RemoveContainer" containerID="c25992503e6b6ff4dd8cdb9d0123835759ee5d7e4a3778fed6fb10015ff84db6"
Nov 03 11:00:20.203168 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:20.203145    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895060-8lw9p_odh-ai-gateway-infra(b0a9bb76-42d9-444e-997b-69edb5081787)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" podUID="b0a9bb76-42d9-444e-997b-69edb5081787"
Nov 03 11:00:20.203647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.179784    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0a9bb76-42d9-444e-997b-69edb5081787" containerID="c25992503e6b6ff4dd8cdb9d0123835759ee5d7e4a3778fed6fb10015ff84db6" exitCode=60
Nov 03 11:00:20.211609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.211243    5817 scope.go:122] "RemoveContainer" containerID="82863d90262a5d4e20af51a894655f5513e2a13f07455b993da129900da48e3c"
Nov 03 11:00:20.213700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.212315    5817 scope.go:122] "RemoveContainer" containerID="dc0df8101da7da0715f7fba8ce80a63cb05dd480c6b6f06b81f6926c3dcf038a"
Nov 03 11:00:20.213700 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:20.212705    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv_odh-ai-gateway-infra(7e9ccf95-7c78-4f2a-8e16-30745b8314c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8"
Nov 03 11:00:20.213700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.213327    5817 scope.go:122] "RemoveContainer" containerID="d85fd83e6b2c86c435d938a852d3972b9f0e05f4df807e2ebf112747a61fa1a9"
Nov 03 11:00:20.213700 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:20.213633    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9_odh-ai-gateway-infra(0a4235c3-e211-4dae-93e7-d43181533579)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" podUID="0a4235c3-e211-4dae-93e7-d43181533579"
Nov 03 11:00:20.238951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.238879    5817 scope.go:122] "RemoveContainer" containerID="a51bddc4290d773e5f60b31b2192bf9f6d8b6d7c61993e59e0d0e59921403171"
Nov 03 11:00:20.240655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.240615    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:20.247230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247199    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-socket\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247241    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-certs\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247288    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033ade0f-f138-4a4b-b2b8-41d02673adab-istiod-ca-cert\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247318    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-credential-socket\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247385    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-token\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247412    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4nvf\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-kube-api-access-q4nvf\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247469    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-envoy\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247608    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-podinfo\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247644    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-data\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247697    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-socket\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247708    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-certs\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.247996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.247929    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-data\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.248206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.248180    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-credential-socket\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.248375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.248353    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033ade0f-f138-4a4b-b2b8-41d02673adab-istiod-ca-cert\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.255877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.253821    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-envoy\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.255877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.253951    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-podinfo\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.256503 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.256479    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-token\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.259545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.259525    5817 scope.go:122] "RemoveContainer" containerID="d03ebe3d3ad0a2963b2d3d9a8599f75ac099a6632e1454e3dfacfcd87f1c547c"
Nov 03 11:00:20.260199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.260164    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4nvf\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-kube-api-access-q4nvf\") pod \"e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") " pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.306796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.306768    5817 scope.go:122] "RemoveContainer" containerID="6c50545b2e204d25b2f7e818f75246871c5905fcb5e42c1849feb525060706a6"
Nov 03 11:00:20.332193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.332165    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"]
Nov 03 11:00:20.340127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.340099    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895060-8qbf5"]
Nov 03 11:00:20.477791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.477735    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:20.529442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.529413    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:20.669886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.668155    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b858d6dd-e7ac-4895-84e2-fea16754cf28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b858d6dd-e7ac-4895-84e2-fea16754cf28-maas-api-service-ca\") pod \"b858d6dd-e7ac-4895-84e2-fea16754cf28\" (UID: \"b858d6dd-e7ac-4895-84e2-fea16754cf28\") "
Nov 03 11:00:20.669886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.668459    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b858d6dd-e7ac-4895-84e2-fea16754cf28-kube-api-access-j8hxd\" (UniqueName: \"kubernetes.io/projected/b858d6dd-e7ac-4895-84e2-fea16754cf28-kube-api-access-j8hxd\") pod \"b858d6dd-e7ac-4895-84e2-fea16754cf28\" (UID: \"b858d6dd-e7ac-4895-84e2-fea16754cf28\") "
Nov 03 11:00:20.671315 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.670854    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b858d6dd-e7ac-4895-84e2-fea16754cf28-maas-api-service-ca" pod "b858d6dd-e7ac-4895-84e2-fea16754cf28" (UID: "b858d6dd-e7ac-4895-84e2-fea16754cf28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:20.671961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.671848    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:20.675389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.675359    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b858d6dd-e7ac-4895-84e2-fea16754cf28-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:20.675848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.675806    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b858d6dd-e7ac-4895-84e2-fea16754cf28-kube-api-access-j8hxd" pod "b858d6dd-e7ac-4895-84e2-fea16754cf28" (UID: "b858d6dd-e7ac-4895-84e2-fea16754cf28"). InnerVolumeSpecName "kube-api-access-j8hxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:20.777784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.777753    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8hxd\" (UniqueName: \"kubernetes.io/projected/b858d6dd-e7ac-4895-84e2-fea16754cf28-kube-api-access-j8hxd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:20.869125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.869092    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"]
Nov 03 11:00:20.870484 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:20.870452    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033ade0f_f138_4a4b_b2b8_41d02673adab.slice/crio-fed679d667487be19e28dad7a8de439c1e61da3f7aad6553b2094f3587431587 WatchSource:0}: Error finding container fed679d667487be19e28dad7a8de439c1e61da3f7aad6553b2094f3587431587: Status 404 returned error can't find the container with id fed679d667487be19e28dad7a8de439c1e61da3f7aad6553b2094f3587431587
Nov 03 11:00:20.873672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.873633    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:20.873794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.873713    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:20.873794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:20.873756    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:21.227484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.227442    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"
Nov 03 11:00:21.227763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.227450    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7" event={"ID":"b858d6dd-e7ac-4895-84e2-fea16754cf28","Type":"ContainerDied","Data":"1b561010180b748679be8c8816e2bad885d859cd9cdac1ca2567343d70813c8f"}
Nov 03 11:00:21.227763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.227659    5817 scope.go:122] "RemoveContainer" containerID="f29359e4551ae3ae01e119e9dbb4079133a4fc4e22e64915af91838db0dd9de0"
Nov 03 11:00:21.247730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.247698    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" event={"ID":"033ade0f-f138-4a4b-b2b8-41d02673adab","Type":"ContainerStarted","Data":"2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b"}
Nov 03 11:00:21.247848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.247759    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" event={"ID":"033ade0f-f138-4a4b-b2b8-41d02673adab","Type":"ContainerStarted","Data":"fed679d667487be19e28dad7a8de439c1e61da3f7aad6553b2094f3587431587"}
Nov 03 11:00:21.252474 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.252441    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"]
Nov 03 11:00:21.255081 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.255054    5817 generic.go:356] "Generic (PLEG): container finished" podID="f04f7770-fc4a-4c82-a644-42f81a284965" containerID="ac10263ca27fc8e9ecb13186d07500a1f75479e2e62e1e824eaa1632be035fb4" exitCode=60
Nov 03 11:00:21.255210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.255087    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" event={"ID":"f04f7770-fc4a-4c82-a644-42f81a284965","Type":"ContainerDied","Data":"ac10263ca27fc8e9ecb13186d07500a1f75479e2e62e1e824eaa1632be035fb4"}
Nov 03 11:00:21.255210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.255150    5817 scope.go:122] "RemoveContainer" containerID="4485475eff14df2e404b0fffbf6c3556e6e6760ae362f71d0fa72bf903bf04e2"
Nov 03 11:00:21.256519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.255853    5817 scope.go:122] "RemoveContainer" containerID="ac10263ca27fc8e9ecb13186d07500a1f75479e2e62e1e824eaa1632be035fb4"
Nov 03 11:00:21.256519 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:21.256260    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk_odh-ai-gateway-infra(f04f7770-fc4a-4c82-a644-42f81a284965)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" podUID="f04f7770-fc4a-4c82-a644-42f81a284965"
Nov 03 11:00:21.283857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.283830    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"]
Nov 03 11:00:21.333631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.333150    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" podStartSLOduration=2.333132288 podStartE2EDuration="2.333132288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:21.27329968 +0000 UTC m=+1851.478881739" watchObservedRunningTime="2026-11-03 11:00:21.333132288 +0000 UTC m=+1851.538714421"
Nov 03 11:00:21.358039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.356105    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"]
Nov 03 11:00:21.368761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.366442    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:21.368971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.368953    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"]
Nov 03 11:00:21.377222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.377157    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895060-w5ss7"]
Nov 03 11:00:21.482723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.480855    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:21.754214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.754190    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:21.799528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.799488    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a9bb76-42d9-444e-997b-69edb5081787-kube-api-access-dmprk\" (UniqueName: \"kubernetes.io/projected/b0a9bb76-42d9-444e-997b-69edb5081787-kube-api-access-dmprk\") pod \"b0a9bb76-42d9-444e-997b-69edb5081787\" (UID: \"b0a9bb76-42d9-444e-997b-69edb5081787\") "
Nov 03 11:00:21.799856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.799834    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a9bb76-42d9-444e-997b-69edb5081787-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a9bb76-42d9-444e-997b-69edb5081787-maas-api-service-ca\") pod \"b0a9bb76-42d9-444e-997b-69edb5081787\" (UID: \"b0a9bb76-42d9-444e-997b-69edb5081787\") "
Nov 03 11:00:21.802662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.800617    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a9bb76-42d9-444e-997b-69edb5081787-maas-api-service-ca" pod "b0a9bb76-42d9-444e-997b-69edb5081787" (UID: "b0a9bb76-42d9-444e-997b-69edb5081787"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:21.802662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.802198    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a9bb76-42d9-444e-997b-69edb5081787-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:21.805615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.803800    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a9bb76-42d9-444e-997b-69edb5081787-kube-api-access-dmprk" pod "b0a9bb76-42d9-444e-997b-69edb5081787" (UID: "b0a9bb76-42d9-444e-997b-69edb5081787"). InnerVolumeSpecName "kube-api-access-dmprk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:21.903743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.903708    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmprk\" (UniqueName: \"kubernetes.io/projected/b0a9bb76-42d9-444e-997b-69edb5081787-kube-api-access-dmprk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:21.905270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.905243    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:21.907734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.907704    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025c2900-5963-4755-9c63-6dba56744a43" path="/var/lib/kubelet/pods/025c2900-5963-4755-9c63-6dba56744a43/volumes"
Nov 03 11:00:21.908790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.908345    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" path="/var/lib/kubelet/pods/b858d6dd-e7ac-4895-84e2-fea16754cf28/volumes"
Nov 03 11:00:21.916905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:21.916742    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:22.004881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.004843    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-openshift-ca\") pod \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") "
Nov 03 11:00:22.005037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.004963    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-maas-api-service-ca\") pod \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\" (UID: \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\") "
Nov 03 11:00:22.005037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.005006    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-kube-api-access-bf6gl\" (UniqueName: \"kubernetes.io/projected/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-kube-api-access-bf6gl\") pod \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") "
Nov 03 11:00:22.005208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.005192    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-xks-ca\") pod \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\" (UID: \"5a24b2a0-d4ad-41dc-981a-5c735541cfa3\") "
Nov 03 11:00:22.006234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.005266    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-kube-api-access-sl2ph\" (UniqueName: \"kubernetes.io/projected/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-kube-api-access-sl2ph\") pod \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\" (UID: \"7e9ccf95-7c78-4f2a-8e16-30745b8314c8\") "
Nov 03 11:00:22.009116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.006511    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-maas-api-service-ca" pod "7e9ccf95-7c78-4f2a-8e16-30745b8314c8" (UID: "7e9ccf95-7c78-4f2a-8e16-30745b8314c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:22.009116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.006828    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-openshift-ca" pod "5a24b2a0-d4ad-41dc-981a-5c735541cfa3" (UID: "5a24b2a0-d4ad-41dc-981a-5c735541cfa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:22.011110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.010660    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-xks-ca" pod "5a24b2a0-d4ad-41dc-981a-5c735541cfa3" (UID: "5a24b2a0-d4ad-41dc-981a-5c735541cfa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:00:22.011110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.010719    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-kube-api-access-bf6gl" pod "5a24b2a0-d4ad-41dc-981a-5c735541cfa3" (UID: "5a24b2a0-d4ad-41dc-981a-5c735541cfa3"). InnerVolumeSpecName "kube-api-access-bf6gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:22.012601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.012404    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-kube-api-access-sl2ph" pod "7e9ccf95-7c78-4f2a-8e16-30745b8314c8" (UID: "7e9ccf95-7c78-4f2a-8e16-30745b8314c8"). InnerVolumeSpecName "kube-api-access-sl2ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:22.081413 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.073125    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:22.090683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.090655    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:22.106486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.106456    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4235c3-e211-4dae-93e7-d43181533579-kube-api-access-trccf\" (UniqueName: \"kubernetes.io/projected/0a4235c3-e211-4dae-93e7-d43181533579-kube-api-access-trccf\") pod \"0a4235c3-e211-4dae-93e7-d43181533579\" (UID: \"0a4235c3-e211-4dae-93e7-d43181533579\") "
Nov 03 11:00:22.107660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.106854    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4235c3-e211-4dae-93e7-d43181533579-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4235c3-e211-4dae-93e7-d43181533579-maas-api-service-ca\") pod \"0a4235c3-e211-4dae-93e7-d43181533579\" (UID: \"0a4235c3-e211-4dae-93e7-d43181533579\") "
Nov 03 11:00:22.107660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.107552    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:22.107660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.107594    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl2ph\" (UniqueName: \"kubernetes.io/projected/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-kube-api-access-sl2ph\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:22.107660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.107611    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:22.107660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.107624    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ccf95-7c78-4f2a-8e16-30745b8314c8-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:22.107660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.107637    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf6gl\" (UniqueName: \"kubernetes.io/projected/5a24b2a0-d4ad-41dc-981a-5c735541cfa3-kube-api-access-bf6gl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:22.108136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.107662    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4235c3-e211-4dae-93e7-d43181533579-maas-api-service-ca" pod "0a4235c3-e211-4dae-93e7-d43181533579" (UID: "0a4235c3-e211-4dae-93e7-d43181533579"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:22.112429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.112396    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4235c3-e211-4dae-93e7-d43181533579-kube-api-access-trccf" pod "0a4235c3-e211-4dae-93e7-d43181533579" (UID: "0a4235c3-e211-4dae-93e7-d43181533579"). InnerVolumeSpecName "kube-api-access-trccf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:22.209123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.209083    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4235c3-e211-4dae-93e7-d43181533579-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:22.209123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.209109    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trccf\" (UniqueName: \"kubernetes.io/projected/0a4235c3-e211-4dae-93e7-d43181533579-kube-api-access-trccf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:22.272656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.272533    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"
Nov 03 11:00:22.272656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.272539    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb" event={"ID":"5a24b2a0-d4ad-41dc-981a-5c735541cfa3","Type":"ContainerDied","Data":"09398a8e538ba394eabe5d872985205cce00051a553c0b67e73d0fe814130bad"}
Nov 03 11:00:22.272880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.272664    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09398a8e538ba394eabe5d872985205cce00051a553c0b67e73d0fe814130bad"
Nov 03 11:00:22.275777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.275749    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv" event={"ID":"7e9ccf95-7c78-4f2a-8e16-30745b8314c8","Type":"ContainerDied","Data":"c5f503efb2dac0dd33706e7fe5053435e919d0a87b02e3fdf6e1ecca9939ef18"}
Nov 03 11:00:22.275971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.275948    5817 scope.go:122] "RemoveContainer" containerID="dc0df8101da7da0715f7fba8ce80a63cb05dd480c6b6f06b81f6926c3dcf038a"
Nov 03 11:00:22.275971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.275897    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"
Nov 03 11:00:22.284246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.284131    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9" event={"ID":"0a4235c3-e211-4dae-93e7-d43181533579","Type":"ContainerDied","Data":"ab42bb8b1fbc8f01fa9c9d8c717202441092cff96a8d22284961de62123d8558"}
Nov 03 11:00:22.284246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.284232    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"
Nov 03 11:00:22.288066 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.288050    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"
Nov 03 11:00:22.288257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.288225    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p" event={"ID":"b0a9bb76-42d9-444e-997b-69edb5081787","Type":"ContainerDied","Data":"d92965e39b4ddf3b754f68a1c776b394f127b41b1428f30221436642ed4005f6"}
Nov 03 11:00:22.299701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.299677    5817 scope.go:122] "RemoveContainer" containerID="d85fd83e6b2c86c435d938a852d3972b9f0e05f4df807e2ebf112747a61fa1a9"
Nov 03 11:00:22.313608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.313508    5817 scope.go:122] "RemoveContainer" containerID="c25992503e6b6ff4dd8cdb9d0123835759ee5d7e4a3778fed6fb10015ff84db6"
Nov 03 11:00:22.316861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.316835    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"]
Nov 03 11:00:22.324774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.324754    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895060-8lw9p"]
Nov 03 11:00:22.330553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.330532    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"]
Nov 03 11:00:22.337566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.337542    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895060-c7grv"]
Nov 03 11:00:22.347616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.347553    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"]
Nov 03 11:00:22.354081 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.354058    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895060-nh8p9"]
Nov 03 11:00:22.371708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.371688    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"]
Nov 03 11:00:22.423689 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:22.423649    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:00:22.490618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.485775    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:22.549180 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.549156    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:22.622089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.616095    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04f7770-fc4a-4c82-a644-42f81a284965-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04f7770-fc4a-4c82-a644-42f81a284965-maas-api-service-ca\") pod \"f04f7770-fc4a-4c82-a644-42f81a284965\" (UID: \"f04f7770-fc4a-4c82-a644-42f81a284965\") "
Nov 03 11:00:22.622089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.616289    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04f7770-fc4a-4c82-a644-42f81a284965-kube-api-access-bs2tr\" (UniqueName: \"kubernetes.io/projected/f04f7770-fc4a-4c82-a644-42f81a284965-kube-api-access-bs2tr\") pod \"f04f7770-fc4a-4c82-a644-42f81a284965\" (UID: \"f04f7770-fc4a-4c82-a644-42f81a284965\") "
Nov 03 11:00:22.622089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.616483    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04f7770-fc4a-4c82-a644-42f81a284965-maas-api-service-ca" pod "f04f7770-fc4a-4c82-a644-42f81a284965" (UID: "f04f7770-fc4a-4c82-a644-42f81a284965"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:22.622089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.616941    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04f7770-fc4a-4c82-a644-42f81a284965-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:22.624793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.624751    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04f7770-fc4a-4c82-a644-42f81a284965-kube-api-access-bs2tr" pod "f04f7770-fc4a-4c82-a644-42f81a284965" (UID: "f04f7770-fc4a-4c82-a644-42f81a284965"). InnerVolumeSpecName "kube-api-access-bs2tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:22.718108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:22.718079    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs2tr\" (UniqueName: \"kubernetes.io/projected/f04f7770-fc4a-4c82-a644-42f81a284965-kube-api-access-bs2tr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:23.099543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.099360    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:23.108302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.108267    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"]
Nov 03 11:00:23.117116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.117080    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403d0a86-qz2nn"]
Nov 03 11:00:23.312690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.312555    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"
Nov 03 11:00:23.312859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.312545    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk" event={"ID":"f04f7770-fc4a-4c82-a644-42f81a284965","Type":"ContainerDied","Data":"e8d11226423d4d35ea5862d9f14a4223e2423e3549cfbbcb6c89df5ac8fb8831"}
Nov 03 11:00:23.312859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.312786    5817 scope.go:122] "RemoveContainer" containerID="ac10263ca27fc8e9ecb13186d07500a1f75479e2e62e1e824eaa1632be035fb4"
Nov 03 11:00:23.324129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.324102    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:23.329997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.329974    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:00:23.381493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.378853    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"]
Nov 03 11:00:23.388200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.388149    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895060-zl4qk"]
Nov 03 11:00:23.676463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.676378    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:23.876998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.876961    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:23.907606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.907557    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4235c3-e211-4dae-93e7-d43181533579" path="/var/lib/kubelet/pods/0a4235c3-e211-4dae-93e7-d43181533579/volumes"
Nov 03 11:00:23.908239 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.908219    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" path="/var/lib/kubelet/pods/7e9ccf95-7c78-4f2a-8e16-30745b8314c8/volumes"
Nov 03 11:00:23.908834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.908815    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d21ae5-8da1-4b48-9280-6470e93d5c17" path="/var/lib/kubelet/pods/85d21ae5-8da1-4b48-9280-6470e93d5c17/volumes"
Nov 03 11:00:23.909769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.909752    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" path="/var/lib/kubelet/pods/b0a9bb76-42d9-444e-997b-69edb5081787/volumes"
Nov 03 11:00:23.910297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:23.910282    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" path="/var/lib/kubelet/pods/f04f7770-fc4a-4c82-a644-42f81a284965/volumes"
Nov 03 11:00:24.038647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.038614    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:24.774752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.774719    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:24.883528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.883492    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"]
Nov 03 11:00:24.884682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884655    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:24.884682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884682    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884699    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884708    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884733    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884740    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884756    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884762    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884773    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884779    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884793    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884799    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884809    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884815    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884830    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884837    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884849    5817 cpu_manager.go:414] "containerMap: removing container" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884855    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884878    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:24.884881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884885    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884907    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884914    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884924    5817 cpu_manager.go:414] "containerMap: removing container" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884931    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884953    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5a24b2a0-d4ad-41dc-981a-5c735541cfa3" containerName="revoke-keys"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884962    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a24b2a0-d4ad-41dc-981a-5c735541cfa3" containerName="revoke-keys"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884975    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884983    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.884998    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885005    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885162    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885175    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885185    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885197    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885208    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885217    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885227    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885235    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="025c2900-5963-4755-9c63-6dba56744a43" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885244    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a24b2a0-d4ad-41dc-981a-5c735541cfa3" containerName="revoke-keys"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885254    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885263    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885275    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885321    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e9598e-3590-4926-b76d-e7e4b83fcf8a" containerName="cleanup"
Nov 03 11:00:24.885783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.885329    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b858d6dd-e7ac-4895-84e2-fea16754cf28" containerName="cleanup"
Nov 03 11:00:24.895847 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.895823    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:24.898654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.898565    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"]
Nov 03 11:00:24.904417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.904380    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-185908\""
Nov 03 11:00:24.927008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.926978    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:24.959665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.959623    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp\" (UID: \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:24.960225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:24.960197    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kpt2\" (UniqueName: \"kubernetes.io/projected/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-kube-api-access-4kpt2\") pod \"payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp\" (UID: \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:25.068692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:25.062369    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kpt2\" (UniqueName: \"kubernetes.io/projected/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-kube-api-access-4kpt2\") pod \"payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp\" (UID: \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:25.083202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:25.075668    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp\" (UID: \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:25.083202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:25.076720    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp\" (UID: \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:25.109266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:25.109226    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kpt2\" (UniqueName: \"kubernetes.io/projected/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-kube-api-access-4kpt2\") pod \"payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp\" (UID: \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:25.225915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:25.225874    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:25.533623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:25.533597    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"]
Nov 03 11:00:25.534927 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:25.534898    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b55d88b_58c7_48ab_b1ee_5016cb9dc329.slice/crio-a96d433410b1060ef5209a719a51d6e4f8ae09d80ed114a9d069cc7aa877b2b3 WatchSource:0}: Error finding container a96d433410b1060ef5209a719a51d6e4f8ae09d80ed114a9d069cc7aa877b2b3: Status 404 returned error can't find the container with id a96d433410b1060ef5209a719a51d6e4f8ae09d80ed114a9d069cc7aa877b2b3
Nov 03 11:00:25.753065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:25.752962    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:26.368107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:26.368058    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp" event={"ID":"0b55d88b-58c7-48ab-b1ee-5016cb9dc329","Type":"ContainerStarted","Data":"9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa"}
Nov 03 11:00:26.368504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:26.368131    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp" event={"ID":"0b55d88b-58c7-48ab-b1ee-5016cb9dc329","Type":"ContainerStarted","Data":"a96d433410b1060ef5209a719a51d6e4f8ae09d80ed114a9d069cc7aa877b2b3"}
Nov 03 11:00:26.368504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:26.368166    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:26.401488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:26.401428    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp" podStartSLOduration=2.401411491 podStartE2EDuration="2.401411491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:26.383821016 +0000 UTC m=+1856.589403070" watchObservedRunningTime="2026-11-03 11:00:26.401411491 +0000 UTC m=+1856.606993541"
Nov 03 11:00:26.409485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:26.409449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:26.637939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:26.636180    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": read tcp 10.133.0.2:44912->10.133.0.207:15021: read: connection reset by peer" start-of-body=
Nov 03 11:00:26.637939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:26.636251    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": read tcp 10.133.0.2:44912->10.133.0.207:15021: read: connection reset by peer"
Nov 03 11:00:27.086276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.084094    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:27.379372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.379293    5817 generic.go:356] "Generic (PLEG): container finished" podID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerID="f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc" exitCode=0
Nov 03 11:00:27.379794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.379364    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" event={"ID":"759a79a3-3e94-4ac4-8435-b92d3835776c","Type":"ContainerDied","Data":"f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc"}
Nov 03 11:00:27.379859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.379846    5817 scope.go:122] "RemoveContainer" containerID="f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc"
Nov 03 11:00:27.381405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.381377    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:27.381485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.381446    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:27.381485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.381478    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.531143    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"]
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532348    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532368    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532387    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532394    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532446    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532454    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532650    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04f7770-fc4a-4c82-a644-42f81a284965" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532668    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9ccf95-7c78-4f2a-8e16-30745b8314c8" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532683    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a9bb76-42d9-444e-997b-69edb5081787" containerName="cleanup"
Nov 03 11:00:27.534599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.532701    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4235c3-e211-4dae-93e7-d43181533579" containerName="cleanup"
Nov 03 11:00:27.539652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.537250    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.541367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.541247    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:00:27.541367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.541340    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:00:27.546695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.543394    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"]
Nov 03 11:00:27.630850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.630755    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/8cf8b517-84e7-453f-8d0a-c228f68d629d-kube-api-access-6ddr4\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.631035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.630999    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.631189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.631169    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.738599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.733962    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.738599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.734122    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.738599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.734208    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/8cf8b517-84e7-453f-8d0a-c228f68d629d-kube-api-access-6ddr4\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.738599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.735858    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.740617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.740216    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.747392 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.747366    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/8cf8b517-84e7-453f-8d0a-c228f68d629d-kube-api-access-6ddr4\") pod \"maas-api-revoke-sub-52613fd0-cl84p\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:27.777354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.776693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:27.864360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:27.864323    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:28.154886 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:28.154857    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf8b517_84e7_453f_8d0a_c228f68d629d.slice/crio-08871788d076c645def905339a06f907ae109c80e2ed47e2ddcfa82288c7bec1 WatchSource:0}: Error finding container 08871788d076c645def905339a06f907ae109c80e2ed47e2ddcfa82288c7bec1: Status 404 returned error can't find the container with id 08871788d076c645def905339a06f907ae109c80e2ed47e2ddcfa82288c7bec1
Nov 03 11:00:28.155207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:28.155186    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"]
Nov 03 11:00:28.406474 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:28.406382    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p" event={"ID":"8cf8b517-84e7-453f-8d0a-c228f68d629d","Type":"ContainerStarted","Data":"08871788d076c645def905339a06f907ae109c80e2ed47e2ddcfa82288c7bec1"}
Nov 03 11:00:28.410885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:28.410852    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" event={"ID":"759a79a3-3e94-4ac4-8435-b92d3835776c","Type":"ContainerStarted","Data":"9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04"}
Nov 03 11:00:28.470922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:28.470874    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:28.570251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:28.570221    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 11:00:28.570636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:28.570611    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:00:28.570710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:28.570674    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 11:00:29.060174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:29.056400    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:29.427809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:29.427714    5817 generic.go:356] "Generic (PLEG): container finished" podID="8cf8b517-84e7-453f-8d0a-c228f68d629d" containerID="9a12dda80f0c07dd821844f4358d7992d82d327a5ac4776ced05a49347380e35" exitCode=0
Nov 03 11:00:29.428257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:29.427798    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p" event={"ID":"8cf8b517-84e7-453f-8d0a-c228f68d629d","Type":"ContainerDied","Data":"9a12dda80f0c07dd821844f4358d7992d82d327a5ac4776ced05a49347380e35"}
Nov 03 11:00:29.569953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:29.569916    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:00:29.570123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:29.569990    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 11:00:29.682815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:29.682712    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:30.546898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.546860    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:30.570446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.569806    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:00:30.570446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.569869    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 11:00:30.739481 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.739288    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:30.834742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.834707    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-openshift-ca\") pod \"8cf8b517-84e7-453f-8d0a-c228f68d629d\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") "
Nov 03 11:00:30.840662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.837518    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-xks-ca\") pod \"8cf8b517-84e7-453f-8d0a-c228f68d629d\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") "
Nov 03 11:00:30.852944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.836133    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-openshift-ca" pod "8cf8b517-84e7-453f-8d0a-c228f68d629d" (UID: "8cf8b517-84e7-453f-8d0a-c228f68d629d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:30.852944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.843101    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf8b517-84e7-453f-8d0a-c228f68d629d-kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/8cf8b517-84e7-453f-8d0a-c228f68d629d-kube-api-access-6ddr4\") pod \"8cf8b517-84e7-453f-8d0a-c228f68d629d\" (UID: \"8cf8b517-84e7-453f-8d0a-c228f68d629d\") "
Nov 03 11:00:30.852944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.843880    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:30.852944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.848732    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-xks-ca" pod "8cf8b517-84e7-453f-8d0a-c228f68d629d" (UID: "8cf8b517-84e7-453f-8d0a-c228f68d629d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:00:30.852944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.851744    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf8b517-84e7-453f-8d0a-c228f68d629d-kube-api-access-6ddr4" pod "8cf8b517-84e7-453f-8d0a-c228f68d629d" (UID: "8cf8b517-84e7-453f-8d0a-c228f68d629d"). InnerVolumeSpecName "kube-api-access-6ddr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:30.947056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.947022    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf8b517-84e7-453f-8d0a-c228f68d629d-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:30.947056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:30.947051    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/8cf8b517-84e7-453f-8d0a-c228f68d629d-kube-api-access-6ddr4\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:31.258054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:31.256977    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:31.450300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:31.450260    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"
Nov 03 11:00:31.450486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:31.450300    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p" event={"ID":"8cf8b517-84e7-453f-8d0a-c228f68d629d","Type":"ContainerDied","Data":"08871788d076c645def905339a06f907ae109c80e2ed47e2ddcfa82288c7bec1"}
Nov 03 11:00:31.450486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:31.450380    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08871788d076c645def905339a06f907ae109c80e2ed47e2ddcfa82288c7bec1"
Nov 03 11:00:31.570422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:31.570344    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:00:31.570422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:31.570398    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 11:00:31.969716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:31.968187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:32.008598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:31.996021    5817 scope.go:122] "RemoveContainer" containerID="d57cf0f6f190f37be4c280d6f2cff65874bdaf065620b35bcc0ea0a1602d3ae9"
Nov 03 11:00:32.051282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:32.051214    5817 scope.go:122] "RemoveContainer" containerID="9467954b4570ca30229c783457171387889d89c6d6a90bc3dc62cd60fa9e33a1"
Nov 03 11:00:32.063065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:32.063045    5817 scope.go:122] "RemoveContainer" containerID="6e6b3d8c938fbfe79968a77fdb591095f130844f4c953de0c9ed649c44e7318c"
Nov 03 11:00:32.073783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:32.073753    5817 scope.go:122] "RemoveContainer" containerID="700482ae49d420f7534273f6b56d69bf38fd406edd505d2537c8abe683ace4c4"
Nov 03 11:00:32.084059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:32.084041    5817 scope.go:122] "RemoveContainer" containerID="48643a71a18f248387593cbf2d66869004bc8a5bc072aaad63acc1f670f1fc16"
Nov 03 11:00:32.126770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:32.126747    5817 scope.go:122] "RemoveContainer" containerID="d6d3d82b5d3bd3ed70ff47225a9ab54f170303b631b987f073b382347fdfc366"
Nov 03 11:00:32.571638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:32.570559    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:00:32.571638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:32.570637    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 11:00:32.588594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:32.585018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:33.276755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:33.274190    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:33.569418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:33.569336    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:00:33.569418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:33.569394    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 11:00:33.965599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:33.963323    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:34.032587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:34.032243    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"]
Nov 03 11:00:34.039822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:34.039787    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bf6ca1-fn7s9"]
Nov 03 11:00:34.574333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:34.574305    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 11:00:34.574851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:34.574831    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 11:00:34.575609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:34.575560    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 11:00:34.633602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:34.631785    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:34.776681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:34.776652    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:34.917158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:34.917063    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:35.542179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:35.542139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:35.904359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:35.904276    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129836b0-55cd-4d36-9dbd-3080928ec67c" path="/var/lib/kubelet/pods/129836b0-55cd-4d36-9dbd-3080928ec67c/volumes"
Nov 03 11:00:36.379904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:36.379861    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:36.959768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:36.959737    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:37.382944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:37.382872    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:00:37.686662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:37.684865    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:38.369732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:38.369693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:39.080838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:39.080796    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:39.581557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:39.581520    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:40.160291 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:40.160254    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:40.879540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:40.879503    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:41.665063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:41.659108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:42.041452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:42.041412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:42.307212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:42.307092    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:43.126719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:43.126681    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:43.254360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:43.254328    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:43.651877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:43.650252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:44.035242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:44.035205    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"]
Nov 03 11:00:44.038752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:44.038729    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c151691e-5lgdd"]
Nov 03 11:00:44.385073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:44.384967    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:45.082596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:45.080190    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:45.847006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:45.846974    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:45.905308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:45.905275    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6621ad-9182-4645-9119-2b7e7c398ff0" path="/var/lib/kubelet/pods/2c6621ad-9182-4645-9119-2b7e7c398ff0/volumes"
Nov 03 11:00:46.526615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:46.523562    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:47.058395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:47.058359    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:47.411649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:47.403396    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:47.614015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:47.613826    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:47.784542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:47.783292    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:48.537195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:48.537089    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:49.095369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:49.093676    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:49.247526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:49.247487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:49.545949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:49.545918    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:49.791405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:49.791351    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:49.965718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:49.965686    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:50.188385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:50.188353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:50.581598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:50.573344    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:50.730408 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:50.730369    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:50.868799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:50.868721    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:51.011991 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.011953    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:51.205184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.205151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:51.656564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.656487    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"]
Nov 03 11:00:51.657249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.657228    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8cf8b517-84e7-453f-8d0a-c228f68d629d" containerName="revoke-keys"
Nov 03 11:00:51.657249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.657244    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf8b517-84e7-453f-8d0a-c228f68d629d" containerName="revoke-keys"
Nov 03 11:00:51.657439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.657359    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf8b517-84e7-453f-8d0a-c228f68d629d" containerName="revoke-keys"
Nov 03 11:00:51.661259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.661235    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.666984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.666958    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-cbeebd-openshift-default-dockercfg-g27jf\""
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.742647    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-data\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.742799    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-certs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.742828    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-token\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.742903    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.743022    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-credential-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.743079    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-envoy\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.743125    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istiod-ca-cert\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.743165    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2pgs\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-kube-api-access-q2pgs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.743809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.743225    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-podinfo\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.758705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.758622    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"]
Nov 03 11:00:51.818161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.818125    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:51.858044 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.858005    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-credential-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.858427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.858365    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-envoy\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.858620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.858604    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istiod-ca-cert\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.858763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.858743    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2pgs\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-kube-api-access-q2pgs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.858934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.858918    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-podinfo\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.859249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.859139    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-data\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.859449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.859429    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-certs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.859584 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.859550    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-token\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.863864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.862246    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istiod-ca-cert\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.863973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.863859    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.863973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.863277    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-credential-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.863973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.862492    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-data\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.864159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.863497    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-certs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.864225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.864162    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.870854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.870805    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-envoy\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.874498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.874461    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-token\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.876547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.876525    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2pgs\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-kube-api-access-q2pgs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.878256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.878226    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-podinfo\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:51.989875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:51.989841    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:52.082563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.082518    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:52.180010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.179796    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:52.425997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.425965    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"]
Nov 03 11:00:52.434533 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:00:52.434500    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:00:52.435112 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:52.435081    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf906dbec_7c86_4e5a_a394_ff8a83fb03c1.slice/crio-f2148c4982a5a43ff11a8132d4d7e9de4e8319a7acf3d29aa7644bfb3fcf3182 WatchSource:0}: Error finding container f2148c4982a5a43ff11a8132d4d7e9de4e8319a7acf3d29aa7644bfb3fcf3182: Status 404 returned error can't find the container with id f2148c4982a5a43ff11a8132d4d7e9de4e8319a7acf3d29aa7644bfb3fcf3182
Nov 03 11:00:52.437871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.437840    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:52.437998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.437920    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:52.437998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.437955    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:52.664372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.664340    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" event={"ID":"f906dbec-7c86-4e5a-a394-ff8a83fb03c1","Type":"ContainerStarted","Data":"10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668"}
Nov 03 11:00:52.664799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.664399    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" event={"ID":"f906dbec-7c86-4e5a-a394-ff8a83fb03c1","Type":"ContainerStarted","Data":"f2148c4982a5a43ff11a8132d4d7e9de4e8319a7acf3d29aa7644bfb3fcf3182"}
Nov 03 11:00:52.685460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:52.685407    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podStartSLOduration=1.685388776 podStartE2EDuration="1.685388776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:52.683140439 +0000 UTC m=+1882.888722491" watchObservedRunningTime="2026-11-03 11:00:52.685388776 +0000 UTC m=+1882.890970830"
Nov 03 11:00:53.082935 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:53.082844    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:53.088928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:53.088901    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:53.349382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:53.347734    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:53.675267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:53.675185    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:53.676352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:53.676332    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:00:53.824792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:53.824760    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:53.965201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:53.965153    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:54.481362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:54.479243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:55.241074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.241037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:55.257269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.257238    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"]
Nov 03 11:00:55.262114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.262077    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:55.273095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.273075    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-917da2\""
Nov 03 11:00:55.273332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.273293    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-917da2-dockercfg-cdql9\""
Nov 03 11:00:55.280304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.280279    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"]
Nov 03 11:00:55.395049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.395007    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r\" (UID: \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:55.395455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.395431    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4rj\" (UniqueName: \"kubernetes.io/projected/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-kube-api-access-xs4rj\") pod \"payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r\" (UID: \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:55.474224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.462359    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:55.498847 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.498755    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r\" (UID: \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:55.499155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.499126    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4rj\" (UniqueName: \"kubernetes.io/projected/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-kube-api-access-xs4rj\") pod \"payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r\" (UID: \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:55.501994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.501410    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r\" (UID: \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:55.551329 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.551286    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4rj\" (UniqueName: \"kubernetes.io/projected/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-kube-api-access-xs4rj\") pod \"payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r\" (UID: \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:55.583875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.583838    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:55.610840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.610805    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:55.882212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:55.882171    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"]
Nov 03 11:00:55.882738 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:55.882713    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fd6984_731f_43d9_98fc_0cfd8f2816fa.slice/crio-e78f5a0a9204fe41f79c01fead88bac8365bac503e908d74766ad202af7c2a4b WatchSource:0}: Error finding container e78f5a0a9204fe41f79c01fead88bac8365bac503e908d74766ad202af7c2a4b: Status 404 returned error can't find the container with id e78f5a0a9204fe41f79c01fead88bac8365bac503e908d74766ad202af7c2a4b
Nov 03 11:00:56.033674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.030162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:56.744947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.744915    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r" event={"ID":"f5fd6984-731f-43d9-98fc-0cfd8f2816fa","Type":"ContainerStarted","Data":"f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b"}
Nov 03 11:00:56.745451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.744970    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r" event={"ID":"f5fd6984-731f-43d9-98fc-0cfd8f2816fa","Type":"ContainerStarted","Data":"e78f5a0a9204fe41f79c01fead88bac8365bac503e908d74766ad202af7c2a4b"}
Nov 03 11:00:56.745451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.745102    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:00:56.767381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.767322    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r" podStartSLOduration=1.7673049920000001 podStartE2EDuration="1.767304992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:56.76196104 +0000 UTC m=+1886.967543093" watchObservedRunningTime="2026-11-03 11:00:56.767304992 +0000 UTC m=+1886.972887043"
Nov 03 11:00:56.775154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.775121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:56.790616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.790590    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"]
Nov 03 11:00:56.803658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.800397    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:56.803961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.803940    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"]
Nov 03 11:00:56.894605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.889199    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cbeebd\""
Nov 03 11:00:56.897914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.897883    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cbeebd-dockercfg-bb2mp\""
Nov 03 11:00:56.939184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.939138    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:56.956599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.953602    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w75m\" (UniqueName: \"kubernetes.io/projected/4fea08a2-6386-4412-a37c-4c75bd7c135e-kube-api-access-7w75m\") pod \"payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs\" (UID: \"4fea08a2-6386-4412-a37c-4c75bd7c135e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:56.956599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:56.953697    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fea08a2-6386-4412-a37c-4c75bd7c135e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs\" (UID: \"4fea08a2-6386-4412-a37c-4c75bd7c135e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:57.066649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.058607    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w75m\" (UniqueName: \"kubernetes.io/projected/4fea08a2-6386-4412-a37c-4c75bd7c135e-kube-api-access-7w75m\") pod \"payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs\" (UID: \"4fea08a2-6386-4412-a37c-4c75bd7c135e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:57.066649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.058702    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fea08a2-6386-4412-a37c-4c75bd7c135e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs\" (UID: \"4fea08a2-6386-4412-a37c-4c75bd7c135e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:57.066649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.060629    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fea08a2-6386-4412-a37c-4c75bd7c135e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs\" (UID: \"4fea08a2-6386-4412-a37c-4c75bd7c135e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:57.066649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.061059    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"]
Nov 03 11:00:57.078595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.071639    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w75m\" (UniqueName: \"kubernetes.io/projected/4fea08a2-6386-4412-a37c-4c75bd7c135e-kube-api-access-7w75m\") pod \"payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs\" (UID: \"4fea08a2-6386-4412-a37c-4c75bd7c135e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:57.105644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.099779    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532dd40-b7f8g"]
Nov 03 11:00:57.125556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.125526    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:57.438659 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:00:57.438630    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fea08a2_6386_4412_a37c_4c75bd7c135e.slice/crio-df1a5362f8f8830dfd764f045cbda542e9f2f8e4cc64bf9b364076f9f754c5c8 WatchSource:0}: Error finding container df1a5362f8f8830dfd764f045cbda542e9f2f8e4cc64bf9b364076f9f754c5c8: Status 404 returned error can't find the container with id df1a5362f8f8830dfd764f045cbda542e9f2f8e4cc64bf9b364076f9f754c5c8
Nov 03 11:00:57.439109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.439086    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"]
Nov 03 11:00:57.729900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.729675    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:57.782273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.782230    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs" event={"ID":"4fea08a2-6386-4412-a37c-4c75bd7c135e","Type":"ContainerStarted","Data":"b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570"}
Nov 03 11:00:57.782995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.782964    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs" event={"ID":"4fea08a2-6386-4412-a37c-4c75bd7c135e","Type":"ContainerStarted","Data":"df1a5362f8f8830dfd764f045cbda542e9f2f8e4cc64bf9b364076f9f754c5c8"}
Nov 03 11:00:57.783658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.783258    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:00:57.800755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.800708    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs" podStartSLOduration=1.8006942339999998 podStartE2EDuration="1.800694234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:57.799892173 +0000 UTC m=+1888.005474225" watchObservedRunningTime="2026-11-03 11:00:57.800694234 +0000 UTC m=+1888.006276286"
Nov 03 11:00:57.908513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:57.908474    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff819c8-689a-4878-b560-9da617880b3f" path="/var/lib/kubelet/pods/aff819c8-689a-4878-b560-9da617880b3f/volumes"
Nov 03 11:00:58.476676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:58.476564    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:59.162122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:59.155121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:00:59.769597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:00:59.764594    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:00.435294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:00.435259    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:01.022600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:01.019604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:01.649619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:01.641183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:02.365599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:02.365296    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:02.993392 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:02.988273    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:03.775413 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:03.775179    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:04.014805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:04.014770    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:04.165794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:04.165707    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:04.385640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:04.385605    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:05.310552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:05.310517    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:05.938126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:05.938094    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:06.088359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:06.088321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:06.862615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:06.862581    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:07.022158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:07.022120    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:07.243949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:07.243890    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:07.381913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:07.381872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:07.786387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:07.786362    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:01:07.976440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:07.976409    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:01:08.246437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:08.246398    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:08.798860 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:08.798832    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:01:09.434906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:09.434828    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:09.572151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:09.572114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:09.844700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:09.844559    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:10.779601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:10.776340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:11.517273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:11.517238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:12.145243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:12.145205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:12.934596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:12.932226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:13.573182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:13.573146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:14.291132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:14.291059    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:14.912623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:14.912133    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:15.029796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:15.029757    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"]
Nov 03 11:01:15.034270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:15.034238    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed60996-hllmp"]
Nov 03 11:01:15.591950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:15.591779    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:15.906062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:15.905977    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aebc73b-ee48-41a3-8989-351f6bac0f5d" path="/var/lib/kubelet/pods/4aebc73b-ee48-41a3-8989-351f6bac0f5d/volumes"
Nov 03 11:01:16.251119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:16.251078    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:16.869300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:16.869053    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:17.515142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:17.515105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:18.154478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:18.154432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:18.830869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:18.830823    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:19.459465 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:19.459392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:20.062494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:20.062463    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:20.199301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:20.199268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:20.371096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:20.371021    5817 generic.go:356] "Generic (PLEG): container finished" podID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerID="2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b" exitCode=0
Nov 03 11:01:20.371258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:20.371091    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" event={"ID":"033ade0f-f138-4a4b-b2b8-41d02673adab","Type":"ContainerDied","Data":"2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b"}
Nov 03 11:01:20.371512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:20.371497    5817 scope.go:122] "RemoveContainer" containerID="2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b"
Nov 03 11:01:20.373182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:20.373157    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:20.373244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:20.373222    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:20.373280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:20.373255    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:21.190599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:21.185811    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:21.382545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:21.382506    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" event={"ID":"033ade0f-f138-4a4b-b2b8-41d02673adab","Type":"ContainerStarted","Data":"3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b"}
Nov 03 11:01:21.494706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:21.485932    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:01:21.494706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:21.488683    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 03 11:01:21.494706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:21.488743    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 03 11:01:21.979602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:21.976494    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:22.037468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:22.037267    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"]
Nov 03 11:01:22.041806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:22.041775    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd31fbcf-psn9m"]
Nov 03 11:01:22.452863 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:01:22.452829    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:01:22.478794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:22.478761    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 03 11:01:22.478945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:22.478815    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 03 11:01:22.653599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:22.648097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:23.333596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:23.323683    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:01:23.431426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:23.431293    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:23.479207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:23.479171    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 03 11:01:23.479624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:23.479230    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 03 11:01:23.561686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:23.561651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:23.920047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:23.920017    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14091ff-a442-44ec-a0f5-00cc4ca4f113" path="/var/lib/kubelet/pods/c14091ff-a442-44ec-a0f5-00cc4ca4f113/volumes"
Nov 03 11:01:24.245597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:24.243164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:24.479193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:24.479114    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 03 11:01:24.479193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:24.479173    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 03 11:01:24.501251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:24.501164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:25.338415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:25.338377    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:25.479021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:25.478982    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 03 11:01:25.479232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:25.479046    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 03 11:01:25.729599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:25.725594    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:26.084547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.084449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:26.227004 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.226963    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:26.478838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.478761    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 03 11:01:26.479140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.478840    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 03 11:01:26.528111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.528077    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"]
Nov 03 11:01:26.535057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.535028    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.560006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.559941    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"]
Nov 03 11:01:26.560707 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.560685    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fb5766-openshift-default-dockercfg-2bbc2\""
Nov 03 11:01:26.614198 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.614173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:26.645439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.645404    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-credential-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.645652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.645472    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-certs\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.645652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.645607    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-data\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.645765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.645661    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.645765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.645749    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c02686-3aa2-4412-8738-4929000230a0-istio-podinfo\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.645869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.645818    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n487\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-kube-api-access-7n487\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.645869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.645849    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-istio-token\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.646005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.645979    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c02686-3aa2-4412-8738-4929000230a0-istiod-ca-cert\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.646070 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.646057    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-envoy\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.753092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.752996    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n487\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-kube-api-access-7n487\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.753720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.753682    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-istio-token\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.761755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.761727    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c02686-3aa2-4412-8738-4929000230a0-istiod-ca-cert\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.761983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.761968    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-envoy\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.762160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.762143    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-credential-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.762316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.762300    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-certs\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.762442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.762429    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-data\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.762591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.762557    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.762776 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.762760    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c02686-3aa2-4412-8738-4929000230a0-istio-podinfo\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.771594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.763736    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-credential-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.793603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.776877    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-certs\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.793603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.757012    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:26.793603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.777195    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-data\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.793603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.777384    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.793603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.770609    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c02686-3aa2-4412-8738-4929000230a0-istiod-ca-cert\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.835913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.833193    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-istio-token\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.836743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.836712    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c02686-3aa2-4412-8738-4929000230a0-istio-podinfo\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.837157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.837134    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-envoy\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.842591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.837373    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n487\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-kube-api-access-7n487\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.866597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.866175    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:26.994152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:26.937390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:27.331387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.331351    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:27.331519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.331439    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:27.331519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.331483    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:27.334522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.334452    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"]
Nov 03 11:01:27.485081 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.485047    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:01:27.486558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.486531    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:01:27.519191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.519153    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" event={"ID":"77c02686-3aa2-4412-8738-4929000230a0","Type":"ContainerStarted","Data":"4ca92cc1cbc4d123120205e6fa3855e9fb5dee81d91e3921cf9b779e970da30b"}
Nov 03 11:01:27.519391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.519208    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" event={"ID":"77c02686-3aa2-4412-8738-4929000230a0","Type":"ContainerStarted","Data":"858429afa70cb066d653e0415d060c047f2c9bb83f02f93119b8cc269c06f077"}
Nov 03 11:01:27.763367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.762176    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:27.867295 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:27.867263    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:28.031458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:28.031327    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" podStartSLOduration=2.031313213 podStartE2EDuration="2.031313213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:27.57040267 +0000 UTC m=+1917.775984736" watchObservedRunningTime="2026-11-03 11:01:28.031313213 +0000 UTC m=+1918.236895268"
Nov 03 11:01:28.042987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:28.042952    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"]
Nov 03 11:01:28.044431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:28.044395    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40a0af9-z8gnn"]
Nov 03 11:01:28.367402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:28.367310    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:28.873907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:28.873877    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:29.361672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:29.349949    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:29.542950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:29.542916    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:29.543851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:29.543832    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:01:29.906124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:29.906094    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa9357a-6d83-496b-bb36-adfb70da00ec" path="/var/lib/kubelet/pods/5aa9357a-6d83-496b-bb36-adfb70da00ec/volumes"
Nov 03 11:01:30.090599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:30.089657    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:30.259107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:30.259068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:30.753855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:30.753816    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:31.333259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:31.333221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:31.488617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:31.488494    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:32.289567 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.289530    5817 scope.go:122] "RemoveContainer" containerID="06be730dec2db25a68f03b564e170bfbfb6328282a2181b65ec19f75ecd4a684"
Nov 03 11:01:32.301391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.301365    5817 scope.go:122] "RemoveContainer" containerID="72e3885803b04e8dcdeafb30092f69949d18cce72ded2e6383ad4a799ff4affa"
Nov 03 11:01:32.313733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.313708    5817 scope.go:122] "RemoveContainer" containerID="93ad65c5bd1f7046c80bd57a52ca7b3862d74f6b52015ad6dbd169f208f8c13e"
Nov 03 11:01:32.331554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.331534    5817 scope.go:122] "RemoveContainer" containerID="7bde258eab4e8692a684df792e4f81082bc0541ed0aeb51970a8290fcc5ad7d0"
Nov 03 11:01:32.341613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.341591    5817 scope.go:122] "RemoveContainer" containerID="b966a7a4530adae64834dfc49665412988ccab84ab1e50fa1aa62e7e842ac5f3"
Nov 03 11:01:32.368541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.366220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:32.409281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.409254    5817 scope.go:122] "RemoveContainer" containerID="d1e110f1119596ee6853b943b23eab48106e737cc358788a049bc054d436627d"
Nov 03 11:01:32.424420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.424387    5817 scope.go:122] "RemoveContainer" containerID="de524aed76e26badeb5a30ef181783bc38e31d3793b8e8f79c8ae291e26b695f"
Nov 03 11:01:32.435901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.435876    5817 scope.go:122] "RemoveContainer" containerID="ffcd4080ab3296e52d6e920fe843b4180cfa5ca5f054e8c63a99af6e995b4e13"
Nov 03 11:01:32.447359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:32.447342    5817 scope.go:122] "RemoveContainer" containerID="b48d042299d17ec2b37a6c765a83bdf903cdcd15b91a3b7ca865a505a52be4ff"
Nov 03 11:01:33.082517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:33.082473    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:33.757303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:33.757270    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:34.443713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:34.443674    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:35.121830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:35.121792    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:35.813100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:35.813059    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:36.714599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:36.714385    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:37.219431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:37.219395    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:37.509255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:37.509174    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:37.659502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:37.659471    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:38.016562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:38.016526    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:38.468274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:38.468238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:39.184841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:39.184808    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:39.777113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:39.777081    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:40.574597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:40.572558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:41.016778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:41.016739    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:41.034859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:41.034833    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"]
Nov 03 11:01:41.039285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:41.039258    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cd9a19e-q9wq6"]
Nov 03 11:01:41.156104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:41.156066    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:41.368145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:41.368056    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:41.596949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:41.596909    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:41.985451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:41.984702    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1" path="/var/lib/kubelet/pods/26f40f5c-ab26-4d7a-b05c-2b5fd3748fb1/volumes"
Nov 03 11:01:42.423134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:42.423054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:42.585411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:42.585374    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:42.825262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:42.825228    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:42.996638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:42.996605    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:43.154042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:43.153951    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:43.433204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:43.433173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:43.597410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:43.597378    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:44.117337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:44.117130    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:44.790468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:44.599122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:44.952285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:44.952250    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:45.183609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:45.183555    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:46.672370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:46.672337    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:47.482118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:47.482083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:47.615908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:47.615876    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:47.947959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:47.947906    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:48.950785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:48.950753    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:49.747282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:49.747242    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:50.473386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:50.473350    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:51.372732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:51.372625    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:52.048691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:52.048659    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:52.413600 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:01:52.413489    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:01:52.682934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:52.680399    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:53.384688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:53.384556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:53.936313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:53.936126    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": read tcp 10.133.0.2:43712->10.133.0.222:15021: read: connection reset by peer" start-of-body=
Nov 03 11:01:53.936313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:53.936208    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": read tcp 10.133.0.2:43712->10.133.0.222:15021: read: connection reset by peer"
Nov 03 11:01:54.094049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.093941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:54.116133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.116104    5817 generic.go:356] "Generic (PLEG): container finished" podID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerID="10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668" exitCode=0
Nov 03 11:01:54.116312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.116178    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" event={"ID":"f906dbec-7c86-4e5a-a394-ff8a83fb03c1","Type":"ContainerDied","Data":"10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668"}
Nov 03 11:01:54.116724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.116705    5817 scope.go:122] "RemoveContainer" containerID="10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668"
Nov 03 11:01:54.118894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.118858    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:54.119021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.118952    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:54.119021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.118997    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:54.663493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.659365    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:54.952595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:54.949252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:55.103266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:55.103231    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:55.134447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:55.134413    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" event={"ID":"f906dbec-7c86-4e5a-a394-ff8a83fb03c1","Type":"ContainerStarted","Data":"f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715"}
Nov 03 11:01:55.457600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:55.457422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:56.083476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:56.083446    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:01:56.083888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:56.083781    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 11:01:56.083888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:56.083839    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 11:01:56.249997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:56.248946    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:56.379758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:56.379666    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:56.968179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:56.968143    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:57.083404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:57.083371    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 11:01:57.083659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:57.083445    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 11:01:57.737171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:57.737130    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:58.125662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:58.112052    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 11:01:58.126271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:58.126231    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 11:01:58.546355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:58.539308    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:01:59.083315 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:59.083282    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 11:01:59.083499 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:59.083354    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 11:01:59.534502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:01:59.525123    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:00.083810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:00.083776    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 11:02:00.083989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:00.083824    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 11:02:00.487103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:00.487068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:00.624820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:00.624759    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:00.745804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:00.745708    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:01.087787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:01.087715    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:02:01.088248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:01.088230    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:02:01.089150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:01.089129    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:02:01.890916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:01.890879    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:02.790598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:02.788215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:03.543823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:03.543779    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:04.186970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:04.186935    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:04.845159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:04.845126    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:05.485804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:05.485756    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:06.449229 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:06.447254    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:07.080501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:07.080401    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:07.358253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:07.358164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:07.542208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:07.542171    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:07.783881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:07.783849    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:08.352202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:08.352167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:09.023285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:09.023252    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"]
Nov 03 11:02:09.027675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:09.027649    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4160bfd-7nqcv"]
Nov 03 11:02:09.177676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:09.177641    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:09.857559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:09.856990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:09.909410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:09.909362    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c66bc84-4b3a-434c-9233-8042710123b9" path="/var/lib/kubelet/pods/6c66bc84-4b3a-434c-9233-8042710123b9/volumes"
Nov 03 11:02:10.496204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:10.496158    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:10.640093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:10.640057    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:10.839686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:10.839535    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:11.392152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:11.392101    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:11.956356 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:11.956321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:12.110410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:12.110378    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:12.347831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:12.347738    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:12.902598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:12.896598    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:13.693601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:13.683252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:14.462689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:14.462181    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:15.146489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:15.146454    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:15.358951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:15.358919    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"]
Nov 03 11:02:15.439109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:15.439077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:16.026550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:16.026515    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:16.397161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:16.397079    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:02:16.769698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:16.769662    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"]
Nov 03 11:02:16.769943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:16.769916    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy" containerID="cri-o://357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806" gracePeriod=30
Nov 03 11:02:16.774756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:16.774731    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"]
Nov 03 11:02:16.774964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:16.774943    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy" containerID="cri-o://29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811" gracePeriod=30
Nov 03 11:02:16.778888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:16.778866    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"]
Nov 03 11:02:16.779086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:16.779041    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy" containerID="cri-o://6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb" gracePeriod=30
Nov 03 11:02:17.080566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:17.080395    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:17.250984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:17.250939    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:18.412827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:18.412788    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:19.788050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:19.788012    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:21.260536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:21.260505    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:22.274595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.274264    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:22.282293 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.281920    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 11:02:22.365470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.362858    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 11:02:22.466255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.466097    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476137    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istiod-ca-cert\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476207    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-token\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476262    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-socket\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476310    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-data\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476336    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-envoy\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476372    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istiod-ca-cert\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476404    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-token\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-token\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476434    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-socket\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.476602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476466    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-envoy\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.476565    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-socket\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477319    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-podinfo\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477366    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-data\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477390    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-kube-api-access-w6vsg\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-kube-api-access-w6vsg\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477412    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-credential-socket\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477441    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-data\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477466    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-token\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-token\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477529    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-credential-socket\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477630    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-kube-api-access-slx65\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-kube-api-access-slx65\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477665    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-podinfo\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477706    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-certs\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477757    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-certs\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477828    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-podinfo\") pod \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\" (UID: \"c1d5fdd1-e359-49dc-93a2-01a188e52f24\") "
Nov 03 11:02:22.481621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.477871    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38216033-6b9e-4efe-8ae7-5e6b947349e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38216033-6b9e-4efe-8ae7-5e6b947349e9-istiod-ca-cert\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.483134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.482219    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-kube-api-access-ltrq2\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-kube-api-access-ltrq2\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.483134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.482274    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-envoy\") pod \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\" (UID: \"dbd4b11e-6392-4182-9b06-9a46cce69f7c\") "
Nov 03 11:02:22.483134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.482313    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-credential-socket\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.483134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.482342    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-certs\") pod \"38216033-6b9e-4efe-8ae7-5e6b947349e9\" (UID: \"38216033-6b9e-4efe-8ae7-5e6b947349e9\") "
Nov 03 11:02:22.483523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.483401    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-certs" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.483704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.483678    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-envoy" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.486316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.483916    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-credential-socket" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.486316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.484292    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38216033-6b9e-4efe-8ae7-5e6b947349e9-istiod-ca-cert" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:22.486316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.484276    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-socket" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.486316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.486195    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istiod-ca-cert" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:22.486709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.486630    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-data" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.493681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.493653    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-certs" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.497611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.496251    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-token" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:22.497805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.497780    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-certs" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.498127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.498107    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-socket" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.498551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.498531    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istiod-ca-cert" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:22.499451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.499167    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-kube-api-access-slx65" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "kube-api-access-slx65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:22.502933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.502911    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-envoy" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.508599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.504177    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-token" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:22.508599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.504378    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-socket" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.508599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.506792    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-envoy" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.508912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.508888    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-credential-socket" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.512378    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-podinfo" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.513192    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-data" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.513403    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-credential-socket" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.513779    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-data" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.514623    5817 generic.go:356] "Generic (PLEG): container finished" podID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerID="6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb" exitCode=0
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.514771    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" event={"ID":"38216033-6b9e-4efe-8ae7-5e6b947349e9","Type":"ContainerDied","Data":"6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb"}
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.514822    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw" event={"ID":"38216033-6b9e-4efe-8ae7-5e6b947349e9","Type":"ContainerDied","Data":"9b6d5af73f75e733678ad80c3499482a084a02b4c8b6dbf6998eb787c9f35079"}
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.514851    5817 scope.go:122] "RemoveContainer" containerID="6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb"
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.515000    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.522097    5817 generic.go:356] "Generic (PLEG): container finished" podID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerID="29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811" exitCode=0
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.522273    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" event={"ID":"dbd4b11e-6392-4182-9b06-9a46cce69f7c","Type":"ContainerDied","Data":"29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811"}
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.522319    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q" event={"ID":"dbd4b11e-6392-4182-9b06-9a46cce69f7c","Type":"ContainerDied","Data":"5d9f0d5cb4ba666888b2c6922f7fc0f131c572639371dba4c439923307d5fe54"}
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.522397    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"
Nov 03 11:02:22.524747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.524708    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-kube-api-access-w6vsg" pod "c1d5fdd1-e359-49dc-93a2-01a188e52f24" (UID: "c1d5fdd1-e359-49dc-93a2-01a188e52f24"). InnerVolumeSpecName "kube-api-access-w6vsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:22.525810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.525501    5817 generic.go:356] "Generic (PLEG): container finished" podID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerID="357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806" exitCode=0
Nov 03 11:02:22.525810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.525540    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" event={"ID":"c1d5fdd1-e359-49dc-93a2-01a188e52f24","Type":"ContainerDied","Data":"357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806"}
Nov 03 11:02:22.525810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.525602    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg" event={"ID":"c1d5fdd1-e359-49dc-93a2-01a188e52f24","Type":"ContainerDied","Data":"67ef6c41e9b3f3e5273251989261b0300e4fe1526f1647491d786ad5df232418"}
Nov 03 11:02:22.525810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.525672    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"
Nov 03 11:02:22.528260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.528225    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-podinfo" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:02:22.534664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.534559    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-token" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:22.535299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.535272    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-kube-api-access-ltrq2" pod "38216033-6b9e-4efe-8ae7-5e6b947349e9" (UID: "38216033-6b9e-4efe-8ae7-5e6b947349e9"). InnerVolumeSpecName "kube-api-access-ltrq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:22.546539 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:02:22.546510    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:02:22.548223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.548173    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-podinfo" pod "dbd4b11e-6392-4182-9b06-9a46cce69f7c" (UID: "dbd4b11e-6392-4182-9b06-9a46cce69f7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:02:22.559593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.556942    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:22.580118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.580061    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"]
Nov 03 11:02:22.583723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.583703    5817 scope.go:122] "RemoveContainer" containerID="6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb"
Nov 03 11:02:22.584500 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:02:22.584364    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb\": container with ID starting with 6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb not found: ID does not exist" containerID="6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb"
Nov 03 11:02:22.584500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.584402    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb"} err="failed to get container status \"6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb\": rpc error: code = NotFound desc = could not find container \"6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb\": container with ID starting with 6ba4ecd7b7dc5ce7327948c2f0728cb6c588aba83e35e7049336815caa5620fb not found: ID does not exist"
Nov 03 11:02:22.584500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.584425    5817 scope.go:122] "RemoveContainer" containerID="29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811"
Nov 03 11:02:22.585327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585297    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slx65\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-kube-api-access-slx65\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585332    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585348    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585363    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585378    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585418    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38216033-6b9e-4efe-8ae7-5e6b947349e9-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585433    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltrq2\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-kube-api-access-ltrq2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585447    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585460    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585475    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585488    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585501    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585515    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585530    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585543    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585557    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585590    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585604    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585619    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585632    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38216033-6b9e-4efe-8ae7-5e6b947349e9-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585644    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585658    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.585677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585672    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6vsg\" (UniqueName: \"kubernetes.io/projected/c1d5fdd1-e359-49dc-93a2-01a188e52f24-kube-api-access-w6vsg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.586202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585686    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.586202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585699    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd4b11e-6392-4182-9b06-9a46cce69f7c-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.586202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585712    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38216033-6b9e-4efe-8ae7-5e6b947349e9-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.586202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.585724    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d5fdd1-e359-49dc-93a2-01a188e52f24-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:22.590503 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.590484    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-7zjjg"]
Nov 03 11:02:22.642319 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.642296    5817 scope.go:122] "RemoveContainer" containerID="a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09"
Nov 03 11:02:22.653766 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.653744    5817 scope.go:122] "RemoveContainer" containerID="29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811"
Nov 03 11:02:22.654028 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:02:22.654009    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811\": container with ID starting with 29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811 not found: ID does not exist" containerID="29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811"
Nov 03 11:02:22.654127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.654042    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811"} err="failed to get container status \"29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811\": rpc error: code = NotFound desc = could not find container \"29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811\": container with ID starting with 29f6c65f0c4574d85eb2d1590c447b1aaef4ccb7454d2709dc53e7a1dcab0811 not found: ID does not exist"
Nov 03 11:02:22.654127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.654067    5817 scope.go:122] "RemoveContainer" containerID="a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09"
Nov 03 11:02:22.654414 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:02:22.654393    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09\": container with ID starting with a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09 not found: ID does not exist" containerID="a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09"
Nov 03 11:02:22.654507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.654421    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09"} err="failed to get container status \"a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09\": rpc error: code = NotFound desc = could not find container \"a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09\": container with ID starting with a48c231927d211ab0a23346dd6e642a50db68c8741e06385d1911d99bebbef09 not found: ID does not exist"
Nov 03 11:02:22.654507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.654467    5817 scope.go:122] "RemoveContainer" containerID="357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806"
Nov 03 11:02:22.664980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.664962    5817 scope.go:122] "RemoveContainer" containerID="357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806"
Nov 03 11:02:22.665233 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:02:22.665211    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806\": container with ID starting with 357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806 not found: ID does not exist" containerID="357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806"
Nov 03 11:02:22.665306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.665250    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806"} err="failed to get container status \"357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806\": rpc error: code = NotFound desc = could not find container \"357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806\": container with ID starting with 357341ffff2fa23cb82feea9425ce4fbb6c87b3d1ee0d4e180744ca02587d806 not found: ID does not exist"
Nov 03 11:02:22.721353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.721323    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:22.867375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.867299    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:22.974708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.974625    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"]
Nov 03 11:02:22.997092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:22.997053    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cvlcw"]
Nov 03 11:02:23.072643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:23.047632    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"]
Nov 03 11:02:23.116258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:23.097971    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-cw94q"]
Nov 03 11:02:23.271293 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:23.271258    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:23.422597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:23.419159    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:24.046595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.036946    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" path="/var/lib/kubelet/pods/38216033-6b9e-4efe-8ae7-5e6b947349e9/volumes"
Nov 03 11:02:24.060596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.048243    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" path="/var/lib/kubelet/pods/c1d5fdd1-e359-49dc-93a2-01a188e52f24/volumes"
Nov 03 11:02:24.060596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.049283    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" path="/var/lib/kubelet/pods/dbd4b11e-6392-4182-9b06-9a46cce69f7c/volumes"
Nov 03 11:02:24.335649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.335522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.422193    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"]
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423372    5817 cpu_manager.go:414] "containerMap: removing container" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423395    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423424    5817 cpu_manager.go:414] "containerMap: removing container" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423432    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423454    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423461    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423491    5817 cpu_manager.go:414] "containerMap: removing container" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423499    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423718    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423738    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="38216033-6b9e-4efe-8ae7-5e6b947349e9" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423752    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d5fdd1-e359-49dc-93a2-01a188e52f24" containerName="istio-proxy"
Nov 03 11:02:24.424402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.423768    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd4b11e-6392-4182-9b06-9a46cce69f7c" containerName="istio-proxy"
Nov 03 11:02:24.435635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.431687    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"]
Nov 03 11:02:24.435635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.431782    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.439763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.438871    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-e2e--b8bff1be-dockercfg-45kcv\""
Nov 03 11:02:24.439763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.438974    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-586b7631a686\""
Nov 03 11:02:24.440662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.440453    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fb5766\""
Nov 03 11:02:24.440662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.440516    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"kube-root-ca.crt\""
Nov 03 11:02:24.440662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.440582    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"openshift-service-ca.crt\""
Nov 03 11:02:24.441099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.441078    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"routing-overlay\""
Nov 03 11:02:24.459771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.459744    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.459964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.459945    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2x2g\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-kube-api-access-w2x2g\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.460063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.460049    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.460123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.460082    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.574596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.563624    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.574596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.563848    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2x2g\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-kube-api-access-w2x2g\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.574596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.563928    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.574596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.563975    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.582677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.580074    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.582677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.581052    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.607916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.604265    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2x2g\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-kube-api-access-w2x2g\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.607916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.604591    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:24.763388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:24.763299    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:25.040910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:25.040758    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:25.095420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:25.090543    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"]
Nov 03 11:02:25.095785 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:02:25.095688    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd0f19a_dc47_4894_a20b_8912e2a6ca0f.slice/crio-a0789fde0572d8aed4a4d36b93c3659613217fcac7d562e182b3f709d0b86034 WatchSource:0}: Error finding container a0789fde0572d8aed4a4d36b93c3659613217fcac7d562e182b3f709d0b86034: Status 404 returned error can't find the container with id a0789fde0572d8aed4a4d36b93c3659613217fcac7d562e182b3f709d0b86034
Nov 03 11:02:25.649395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:25.649347    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q" event={"ID":"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f","Type":"ContainerStarted","Data":"b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008"}
Nov 03 11:02:25.649904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:25.649429    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q" event={"ID":"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f","Type":"ContainerStarted","Data":"a0789fde0572d8aed4a4d36b93c3659613217fcac7d562e182b3f709d0b86034"}
Nov 03 11:02:25.649904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:25.649591    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:25.667483 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:25.667432    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q" podStartSLOduration=1.667418617 podStartE2EDuration="1.667418617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:02:25.664418208 +0000 UTC m=+1975.870000260" watchObservedRunningTime="2026-11-03 11:02:25.667418617 +0000 UTC m=+1975.873000668"
Nov 03 11:02:25.912827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:25.912742    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:26.765080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:26.765041    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:27.388879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.384775    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"]
Nov 03 11:02:27.394038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.394006    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.397367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.397336    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"]
Nov 03 11:02:27.407633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.407534    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9f53bb-openshift-default-dockercfg-svbc9\""
Nov 03 11:02:27.462560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.462529    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:27.563923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.563881    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-envoy\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.564333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.564310    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-podinfo\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.564531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.564514    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-credential-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.564725 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.564708    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x49dt\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-kube-api-access-x49dt\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.564890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.564874    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.565062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.565047    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istiod-ca-cert\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.565271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.565254    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-data\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.565513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.565498    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-certs\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.565792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.565772    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-token\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.600112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.600036    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688199    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-data\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688339    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-certs\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688479    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-token\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688549    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-envoy\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688667    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-podinfo\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688724    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-credential-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688753    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x49dt\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-kube-api-access-x49dt\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688793    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.688856    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istiod-ca-cert\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.689964    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istiod-ca-cert\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.690645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.690262    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-data\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.713602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.707878    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-certs\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.713602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.712869    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.713602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.713106    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-credential-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.735352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.726452    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-token\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.752594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.750066    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-podinfo\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.778596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.767827    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x49dt\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-kube-api-access-x49dt\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.778596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.768188    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-envoy\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:27.778596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:27.769785    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:28.034907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.034814    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:28.349590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.349407    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"]
Nov 03 11:02:28.356637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.352679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:28.366401 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:02:28.366330    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129748c3_369d_4a33_9cda_7f5ebabf7d7e.slice/crio-c666034d1cb691c0ad2354be613b3072a57303c4aceab0431278241bc21d1f1b WatchSource:0}: Error finding container c666034d1cb691c0ad2354be613b3072a57303c4aceab0431278241bc21d1f1b: Status 404 returned error can't find the container with id c666034d1cb691c0ad2354be613b3072a57303c4aceab0431278241bc21d1f1b
Nov 03 11:02:28.370979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.370935    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:28.371098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.371029    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:28.371098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.371074    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:28.749790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.749754    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" event={"ID":"129748c3-369d-4a33-9cda-7f5ebabf7d7e","Type":"ContainerStarted","Data":"878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e"}
Nov 03 11:02:28.749967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.749840    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" event={"ID":"129748c3-369d-4a33-9cda-7f5ebabf7d7e","Type":"ContainerStarted","Data":"c666034d1cb691c0ad2354be613b3072a57303c4aceab0431278241bc21d1f1b"}
Nov 03 11:02:28.780663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.780616    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" podStartSLOduration=1.780565348 podStartE2EDuration="1.780565348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:02:28.771260177 +0000 UTC m=+1978.976842236" watchObservedRunningTime="2026-11-03 11:02:28.780565348 +0000 UTC m=+1978.986147392"
Nov 03 11:02:28.953627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:28.950181    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:29.035747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.035657    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:29.044869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.044837    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:29.135260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.135212    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"]
Nov 03 11:02:29.137855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.135437    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy" containerID="cri-o://8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e" gracePeriod=30
Nov 03 11:02:29.544807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.544773    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 11:02:29.545011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.544834    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 11:02:29.760418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.760379    5817 generic.go:356] "Generic (PLEG): container finished" podID="77c02686-3aa2-4412-8738-4929000230a0" containerID="4ca92cc1cbc4d123120205e6fa3855e9fb5dee81d91e3921cf9b779e970da30b" exitCode=0
Nov 03 11:02:29.760675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.760448    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" event={"ID":"77c02686-3aa2-4412-8738-4929000230a0","Type":"ContainerDied","Data":"4ca92cc1cbc4d123120205e6fa3855e9fb5dee81d91e3921cf9b779e970da30b"}
Nov 03 11:02:29.760905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.760884    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:29.760964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.760920    5817 scope.go:122] "RemoveContainer" containerID="4ca92cc1cbc4d123120205e6fa3855e9fb5dee81d91e3921cf9b779e970da30b"
Nov 03 11:02:29.761868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.761853    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:02:29.762696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.762672    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:29.762758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.762737    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:29.762799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.762773    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:29.869634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:29.860549    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:30.381853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.381807    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8" podUID="568dfe2b-947c-446f-9eea-3a4262876815" containerName="main" containerID="cri-o://0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9" gracePeriod=45
Nov 03 11:02:30.799408 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.797790    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:30.822630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.811878    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 11:02:30.898155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.898118    5817 generic.go:356] "Generic (PLEG): container finished" podID="568dfe2b-947c-446f-9eea-3a4262876815" containerID="0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9" exitCode=0
Nov 03 11:02:30.902841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.902809    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8" event={"ID":"568dfe2b-947c-446f-9eea-3a4262876815","Type":"ContainerDied","Data":"0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9"}
Nov 03 11:02:30.903194 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.903172    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8" event={"ID":"568dfe2b-947c-446f-9eea-3a4262876815","Type":"ContainerDied","Data":"3285c073a0d5c91bbbdc2d39a2ea901376379b7b7172695175b0d0296862cdf5"}
Nov 03 11:02:30.903337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.903322    5817 scope.go:122] "RemoveContainer" containerID="0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9"
Nov 03 11:02:30.903528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.902161    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"
Nov 03 11:02:30.914751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.914724    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" event={"ID":"77c02686-3aa2-4412-8738-4929000230a0","Type":"ContainerStarted","Data":"8241c5a81c01d121196d614163c43c2d884341897c0efad0076b5e82537301dc"}
Nov 03 11:02:30.939850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.936829    5817 scope.go:122] "RemoveContainer" containerID="0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9"
Nov 03 11:02:30.939850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.938908    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568dfe2b-947c-446f-9eea-3a4262876815-kube-api-access-gqxrf\" (UniqueName: \"kubernetes.io/projected/568dfe2b-947c-446f-9eea-3a4262876815-kube-api-access-gqxrf\") pod \"568dfe2b-947c-446f-9eea-3a4262876815\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") "
Nov 03 11:02:30.939850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.938968    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568dfe2b-947c-446f-9eea-3a4262876815-tls-certs\" (UniqueName: \"kubernetes.io/secret/568dfe2b-947c-446f-9eea-3a4262876815-tls-certs\") pod \"568dfe2b-947c-446f-9eea-3a4262876815\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") "
Nov 03 11:02:30.939850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.939071    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-model-cache\") pod \"568dfe2b-947c-446f-9eea-3a4262876815\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") "
Nov 03 11:02:30.939850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.939299    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-tmp-dir\") pod \"568dfe2b-947c-446f-9eea-3a4262876815\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") "
Nov 03 11:02:30.939850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.939359    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-dshm\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-dshm\") pod \"568dfe2b-947c-446f-9eea-3a4262876815\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") "
Nov 03 11:02:30.941629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.941560    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-tmp-dir" pod "568dfe2b-947c-446f-9eea-3a4262876815" (UID: "568dfe2b-947c-446f-9eea-3a4262876815"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:30.941772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.941660    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-model-cache" pod "568dfe2b-947c-446f-9eea-3a4262876815" (UID: "568dfe2b-947c-446f-9eea-3a4262876815"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:30.941772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.941754    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-home\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-home\") pod \"568dfe2b-947c-446f-9eea-3a4262876815\" (UID: \"568dfe2b-947c-446f-9eea-3a4262876815\") "
Nov 03 11:02:30.942024 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:02:30.942001    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9\": container with ID starting with 0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9 not found: ID does not exist" containerID="0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9"
Nov 03 11:02:30.942136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.942113    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9"} err="failed to get container status \"0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9\": rpc error: code = NotFound desc = could not find container \"0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9\": container with ID starting with 0ef262d3f482b124e41b94a76bb5256b837146bbdd86a250451ddf380e685bb9 not found: ID does not exist"
Nov 03 11:02:30.942409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.942355    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:30.942409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.942377    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:30.946070 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.944722    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-home" pod "568dfe2b-947c-446f-9eea-3a4262876815" (UID: "568dfe2b-947c-446f-9eea-3a4262876815"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:30.949713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.946268    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568dfe2b-947c-446f-9eea-3a4262876815-kube-api-access-gqxrf" pod "568dfe2b-947c-446f-9eea-3a4262876815" (UID: "568dfe2b-947c-446f-9eea-3a4262876815"). InnerVolumeSpecName "kube-api-access-gqxrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:30.949713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.946546    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-dshm" pod "568dfe2b-947c-446f-9eea-3a4262876815" (UID: "568dfe2b-947c-446f-9eea-3a4262876815"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:30.949713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:30.946972    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568dfe2b-947c-446f-9eea-3a4262876815-tls-certs" pod "568dfe2b-947c-446f-9eea-3a4262876815" (UID: "568dfe2b-947c-446f-9eea-3a4262876815"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:02:31.044556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.044479    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568dfe2b-947c-446f-9eea-3a4262876815-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:31.044556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.044511    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:31.044556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.044526    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568dfe2b-947c-446f-9eea-3a4262876815-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:31.044556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.044539    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqxrf\" (UniqueName: \"kubernetes.io/projected/568dfe2b-947c-446f-9eea-3a4262876815-kube-api-access-gqxrf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:31.223381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.223290    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:31.250335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.250304    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"]
Nov 03 11:02:31.254357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.254333    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8"]
Nov 03 11:02:31.368881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.368791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:31.661108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.660525    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:31.745488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.745452    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-e11de4/auth-test-model-e11de4-kserve-6866b8749-j7cj8" podUID="568dfe2b-947c-446f-9eea-3a4262876815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:02:31.835853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.831144    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:31.866867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.866834    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:02:31.867054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.867036    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 11:02:31.867108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.867082    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 11:02:31.933247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:31.933209    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568dfe2b-947c-446f-9eea-3a4262876815" path="/var/lib/kubelet/pods/568dfe2b-947c-446f-9eea-3a4262876815/volumes"
Nov 03 11:02:32.390721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.387840    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"]
Nov 03 11:02:32.390721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.388940    5817 cpu_manager.go:414] "containerMap: removing container" podUID="568dfe2b-947c-446f-9eea-3a4262876815" containerName="main"
Nov 03 11:02:32.390721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.388959    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568dfe2b-947c-446f-9eea-3a4262876815" containerName="main"
Nov 03 11:02:32.390721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.389196    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="568dfe2b-947c-446f-9eea-3a4262876815" containerName="main"
Nov 03 11:02:32.417072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.417040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:32.440085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.440057    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"]
Nov 03 11:02:32.440276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.440161    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:32.444280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.444259    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9f53bb-dockercfg-jxvml\""
Nov 03 11:02:32.471393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.471369    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9f53bb\""
Nov 03 11:02:32.473224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.473201    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c285b119-1c17-4198-bf9c-07efc70bb500-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv\" (UID: \"c285b119-1c17-4198-bf9c-07efc70bb500\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:32.473444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.473420    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9cp\" (UniqueName: \"kubernetes.io/projected/c285b119-1c17-4198-bf9c-07efc70bb500-kube-api-access-jg9cp\") pod \"payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv\" (UID: \"c285b119-1c17-4198-bf9c-07efc70bb500\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:32.593597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.586069    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9cp\" (UniqueName: \"kubernetes.io/projected/c285b119-1c17-4198-bf9c-07efc70bb500-kube-api-access-jg9cp\") pod \"payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv\" (UID: \"c285b119-1c17-4198-bf9c-07efc70bb500\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:32.593597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.586328    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c285b119-1c17-4198-bf9c-07efc70bb500-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv\" (UID: \"c285b119-1c17-4198-bf9c-07efc70bb500\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:32.593597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.587734    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c285b119-1c17-4198-bf9c-07efc70bb500-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv\" (UID: \"c285b119-1c17-4198-bf9c-07efc70bb500\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:32.619599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.619105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:32.645669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.643711    5817 scope.go:122] "RemoveContainer" containerID="b40b9274ebf454fdbde84b7ba6e60fc62eb5b3fc7532b2ca3d18faa08e3d2bdb"
Nov 03 11:02:32.730598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.711167    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9cp\" (UniqueName: \"kubernetes.io/projected/c285b119-1c17-4198-bf9c-07efc70bb500-kube-api-access-jg9cp\") pod \"payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv\" (UID: \"c285b119-1c17-4198-bf9c-07efc70bb500\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:32.781900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.773242    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:32.812871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.807168    5817 scope.go:122] "RemoveContainer" containerID="f6dbbe065832a171cc3b377bb67ed470b13fe10ec809a189a88ff2e46d583671"
Nov 03 11:02:32.868133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.867709    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 11:02:32.868133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.867765    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 11:02:32.931047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.931017    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:32.942027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.941808    5817 scope.go:122] "RemoveContainer" containerID="d248f6500751e4cdbb46e83099a75947618d3feba71a0ecbad4a5ea3c70324e9"
Nov 03 11:02:32.960597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.960509    5817 scope.go:122] "RemoveContainer" containerID="e64c890afaa269f42f31184a6e3f967631374abfb98359db850628b6a54d4d2c"
Nov 03 11:02:32.978017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:32.977994    5817 scope.go:122] "RemoveContainer" containerID="f9ef28ac2562769a5b20c8712fa178ca778818f567b74c8ea65489f6718a265b"
Nov 03 11:02:33.027287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:33.026779    5817 scope.go:122] "RemoveContainer" containerID="c9be68fa9c5e6db92cab48a4024b722c7e31015f30899e002b67e2f891c23adf"
Nov 03 11:02:33.070493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:33.070398    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:33.181000 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:02:33.180972    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc285b119_1c17_4198_bf9c_07efc70bb500.slice/crio-eec5aa12b1ef553cbf9f36a970f09526ccfec906bb595f1ffc379df682e6dd4a WatchSource:0}: Error finding container eec5aa12b1ef553cbf9f36a970f09526ccfec906bb595f1ffc379df682e6dd4a: Status 404 returned error can't find the container with id eec5aa12b1ef553cbf9f36a970f09526ccfec906bb595f1ffc379df682e6dd4a
Nov 03 11:02:33.181309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:33.181268    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"]
Nov 03 11:02:33.249622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:33.243042    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:33.481544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:33.481500    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:33.632595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:33.632257    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:33.867241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:33.867151    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 11:02:33.867241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:33.867211    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 11:02:34.013595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.012036    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv" event={"ID":"c285b119-1c17-4198-bf9c-07efc70bb500","Type":"ContainerStarted","Data":"6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed"}
Nov 03 11:02:34.013595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.012098    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv" event={"ID":"c285b119-1c17-4198-bf9c-07efc70bb500","Type":"ContainerStarted","Data":"eec5aa12b1ef553cbf9f36a970f09526ccfec906bb595f1ffc379df682e6dd4a"}
Nov 03 11:02:34.013595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.012130    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:34.047595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.037086    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv" podStartSLOduration=2.037066784 podStartE2EDuration="2.037066784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:02:34.03663754 +0000 UTC m=+1984.242219593" watchObservedRunningTime="2026-11-03 11:02:34.037066784 +0000 UTC m=+1984.242648837"
Nov 03 11:02:34.481064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.481037    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 11:02:34.533605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.533531    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-podinfo\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.534607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.533899    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-token\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-token\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.534607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.533944    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-envoy\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.534607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.533982    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-socket\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.534607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.534034    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-data\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.534607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.534098    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1738b7-8733-48da-b0d7-5f3be1475436-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1738b7-8733-48da-b0d7-5f3be1475436-istiod-ca-cert\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.534607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.534240    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-certs\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.534607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.534350    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-credential-socket\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.535049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.534642    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-socket" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:34.535113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535090    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-credential-socket" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:34.535173 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535152    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-data" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:34.535222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535205    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-certs" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:34.535548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535515    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1738b7-8733-48da-b0d7-5f3be1475436-istiod-ca-cert" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:34.535908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535885    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.536014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535914    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.536014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535929    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.536014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535941    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.536014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.535953    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1738b7-8733-48da-b0d7-5f3be1475436-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.540947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.540914    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-podinfo" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:02:34.541090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.541066    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-envoy" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:34.542929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.542908    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-token" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:34.637247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.637207    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-kube-api-access-8xgwm\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-kube-api-access-8xgwm\") pod \"bf1738b7-8733-48da-b0d7-5f3be1475436\" (UID: \"bf1738b7-8733-48da-b0d7-5f3be1475436\") "
Nov 03 11:02:34.638038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.638014    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.638149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.638044    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.638149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.638058    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1738b7-8733-48da-b0d7-5f3be1475436-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.639611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.639549    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-kube-api-access-8xgwm" pod "bf1738b7-8733-48da-b0d7-5f3be1475436" (UID: "bf1738b7-8733-48da-b0d7-5f3be1475436"). InnerVolumeSpecName "kube-api-access-8xgwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:34.689607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.687122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:34.744591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.742280    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xgwm\" (UniqueName: \"kubernetes.io/projected/bf1738b7-8733-48da-b0d7-5f3be1475436-kube-api-access-8xgwm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:34.867580 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.867548    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 11:02:34.867761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:34.867624    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 11:02:35.029928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.029837    5817 generic.go:356] "Generic (PLEG): container finished" podID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerID="8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e" exitCode=0
Nov 03 11:02:35.029928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.029897    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"
Nov 03 11:02:35.030437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.029924    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" event={"ID":"bf1738b7-8733-48da-b0d7-5f3be1475436","Type":"ContainerDied","Data":"8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e"}
Nov 03 11:02:35.030437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.029994    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd" event={"ID":"bf1738b7-8733-48da-b0d7-5f3be1475436","Type":"ContainerDied","Data":"fb5907738945f5e9c2a967d1a3eb38b15b8f0d6b33cc9554ff2a19c87f913594"}
Nov 03 11:02:35.030437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.030023    5817 scope.go:122] "RemoveContainer" containerID="8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e"
Nov 03 11:02:35.043000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.042983    5817 scope.go:122] "RemoveContainer" containerID="8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e"
Nov 03 11:02:35.043278 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:02:35.043258    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e\": container with ID starting with 8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e not found: ID does not exist" containerID="8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e"
Nov 03 11:02:35.043346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.043286    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e"} err="failed to get container status \"8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e\": rpc error: code = NotFound desc = could not find container \"8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e\": container with ID starting with 8d4e9b8e640f62060d8b383bbb6c1b28b97d1d0377d39aaaa8027894b6d1731e not found: ID does not exist"
Nov 03 11:02:35.067699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.067664    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"]
Nov 03 11:02:35.074015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.073987    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-k8gdd"]
Nov 03 11:02:35.367412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.365399    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:35.867227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.867193    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 11:02:35.867420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.867242    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 11:02:35.935959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:35.935132    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" path="/var/lib/kubelet/pods/bf1738b7-8733-48da-b0d7-5f3be1475436/volumes"
Nov 03 11:02:36.047429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:36.047220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:36.663455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:36.663429    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:02:36.819801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:36.819756    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:36.873944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:36.873914    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:02:36.875483 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:36.874432    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:02:36.875483 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:36.875444    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:02:37.671330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:37.671298    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:38.107342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:38.107280    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:38.369257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:38.367725    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:38.953287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:38.953256    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:39.647436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:39.647397    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:40.273609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:40.265684    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:40.953191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:40.949198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:41.690560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:41.690521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:42.470686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:42.470652    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:43.041109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:43.041078    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:43.194317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:43.194283    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:43.653363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:43.653327    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:43.799075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:43.799036    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:43.976446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:43.976413    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:44.231965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:44.231883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:44.392637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:44.392388    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:44.758521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:44.758488    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:45.061836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:45.061755    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:02:45.203474 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:45.203419    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:45.960887 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:45.960857    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:46.595619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:46.595565    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:47.229954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:47.229918    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:47.529403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:47.529320    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:48.013131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:48.013100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:48.165912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:48.165876    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:48.945062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:48.945024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:49.665332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:49.665288    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:50.380363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:50.380319    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:51.045885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:51.045789    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:51.653175 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:51.653132    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:52.361352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:52.361315    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:52.420880 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:02:52.420838    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:02:52.963671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:52.962037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:53.565885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:53.565848    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:53.691777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:53.691746    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:02:54.414160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:54.414117    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:55.523822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:55.523761    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:56.326495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:56.326455    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:57.193239 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:57.193200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:57.649019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:57.648916    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:57.988757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:57.984934    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:58.663938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:58.663904    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:02:59.357995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:02:59.357916    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:00.070806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:00.070771    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:00.738626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:00.738588    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:00.992024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:00.991930    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"]
Nov 03 11:03:00.993162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:00.993142    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy"
Nov 03 11:03:00.993218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:00.993165    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy"
Nov 03 11:03:00.993375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:00.993363    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1738b7-8733-48da-b0d7-5f3be1475436" containerName="istio-proxy"
Nov 03 11:03:00.998690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:00.998663    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.000916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.000893    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"]
Nov 03 11:03:01.005330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.005308    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:03:01.005467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.005341    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:03:01.171873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.171840    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.172388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.172138    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjb6f\" (UniqueName: \"kubernetes.io/projected/5e752beb-cf06-4255-b7ff-01f0d73c7088-kube-api-access-qjb6f\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.172388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.172260    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.274470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.274364    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.274689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.274500    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.274782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.274761    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjb6f\" (UniqueName: \"kubernetes.io/projected/5e752beb-cf06-4255-b7ff-01f0d73c7088-kube-api-access-qjb6f\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.275248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.275223    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.277197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.277173    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.282849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.282815    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjb6f\" (UniqueName: \"kubernetes.io/projected/5e752beb-cf06-4255-b7ff-01f0d73c7088-kube-api-access-qjb6f\") pod \"maas-api-revoke-sub-7cf41438-4llbr\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.322281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.322250    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:01.498375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.497355    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:01.608905 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:03:01.608872    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e752beb_cf06_4255_b7ff_01f0d73c7088.slice/crio-e01613f9d72a6c103d0756d5ad675573e17d9da136cf5154f8fa7d80c9d4a28f WatchSource:0}: Error finding container e01613f9d72a6c103d0756d5ad675573e17d9da136cf5154f8fa7d80c9d4a28f: Status 404 returned error can't find the container with id e01613f9d72a6c103d0756d5ad675573e17d9da136cf5154f8fa7d80c9d4a28f
Nov 03 11:03:01.609119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:01.609093    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"]
Nov 03 11:03:02.125280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:02.125168    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:02.334941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:02.334895    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr" event={"ID":"5e752beb-cf06-4255-b7ff-01f0d73c7088","Type":"ContainerStarted","Data":"56a3d3f470cff61ebc197f9eca059fa8a32242413a91e7230ca79f9a7a3b476d"}
Nov 03 11:03:02.335352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:02.334971    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr" event={"ID":"5e752beb-cf06-4255-b7ff-01f0d73c7088","Type":"ContainerStarted","Data":"e01613f9d72a6c103d0756d5ad675573e17d9da136cf5154f8fa7d80c9d4a28f"}
Nov 03 11:03:02.351853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:02.351807    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr" podStartSLOduration=2.351792151 podStartE2EDuration="2.351792151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:02.350667246 +0000 UTC m=+2012.556249298" watchObservedRunningTime="2026-11-03 11:03:02.351792151 +0000 UTC m=+2012.557374228"
Nov 03 11:03:02.765598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:02.761439    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:03.345753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:03.345713    5817 generic.go:356] "Generic (PLEG): container finished" podID="5e752beb-cf06-4255-b7ff-01f0d73c7088" containerID="56a3d3f470cff61ebc197f9eca059fa8a32242413a91e7230ca79f9a7a3b476d" exitCode=0
Nov 03 11:03:03.346167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:03.345801    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr" event={"ID":"5e752beb-cf06-4255-b7ff-01f0d73c7088","Type":"ContainerDied","Data":"56a3d3f470cff61ebc197f9eca059fa8a32242413a91e7230ca79f9a7a3b476d"}
Nov 03 11:03:03.468666 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:03.468629    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:04.140974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.140938    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:04.535306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.535281    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:04.553379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.553351    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e752beb-cf06-4255-b7ff-01f0d73c7088-kube-api-access-qjb6f\" (UniqueName: \"kubernetes.io/projected/5e752beb-cf06-4255-b7ff-01f0d73c7088-kube-api-access-qjb6f\") pod \"5e752beb-cf06-4255-b7ff-01f0d73c7088\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") "
Nov 03 11:03:04.554462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.553447    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-xks-ca\") pod \"5e752beb-cf06-4255-b7ff-01f0d73c7088\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") "
Nov 03 11:03:04.554462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.553545    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-openshift-ca\") pod \"5e752beb-cf06-4255-b7ff-01f0d73c7088\" (UID: \"5e752beb-cf06-4255-b7ff-01f0d73c7088\") "
Nov 03 11:03:04.554462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.553920    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-openshift-ca" pod "5e752beb-cf06-4255-b7ff-01f0d73c7088" (UID: "5e752beb-cf06-4255-b7ff-01f0d73c7088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:03:04.554462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.554422    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:04.556867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.556841    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-xks-ca" pod "5e752beb-cf06-4255-b7ff-01f0d73c7088" (UID: "5e752beb-cf06-4255-b7ff-01f0d73c7088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:03:04.560094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.560062    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e752beb-cf06-4255-b7ff-01f0d73c7088-kube-api-access-qjb6f" pod "5e752beb-cf06-4255-b7ff-01f0d73c7088" (UID: "5e752beb-cf06-4255-b7ff-01f0d73c7088"). InnerVolumeSpecName "kube-api-access-qjb6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:03:04.655948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.655912    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e752beb-cf06-4255-b7ff-01f0d73c7088-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:04.655948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.655941    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjb6f\" (UniqueName: \"kubernetes.io/projected/5e752beb-cf06-4255-b7ff-01f0d73c7088-kube-api-access-qjb6f\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:04.762114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:04.761587    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:05.438617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:05.436847    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr" event={"ID":"5e752beb-cf06-4255-b7ff-01f0d73c7088","Type":"ContainerDied","Data":"e01613f9d72a6c103d0756d5ad675573e17d9da136cf5154f8fa7d80c9d4a28f"}
Nov 03 11:03:05.438617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:05.436910    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e01613f9d72a6c103d0756d5ad675573e17d9da136cf5154f8fa7d80c9d4a28f"
Nov 03 11:03:05.438617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:05.436980    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"
Nov 03 11:03:05.710653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:05.710006    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:06.506178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:06.505307    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:07.194164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:07.194097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:07.608986 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:07.608896    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:07.892803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:07.892698    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:08.506842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:08.506807    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:09.210784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:09.210736    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:09.887750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:09.887711    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:10.572117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:10.572081    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:11.219465 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.219408    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"]
Nov 03 11:03:11.220659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.220634    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5e752beb-cf06-4255-b7ff-01f0d73c7088" containerName="revoke-keys"
Nov 03 11:03:11.220659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.220658    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e752beb-cf06-4255-b7ff-01f0d73c7088" containerName="revoke-keys"
Nov 03 11:03:11.220893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.220880    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e752beb-cf06-4255-b7ff-01f0d73c7088" containerName="revoke-keys"
Nov 03 11:03:11.225692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.225668    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.230961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.230933    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"]
Nov 03 11:03:11.235544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.235514    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:03:11.236790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.236767    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:03:11.302942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.302907    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.303100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.302964    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6q7\" (UniqueName: \"kubernetes.io/projected/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-kube-api-access-8d6q7\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.303183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.303162    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.405250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.405206    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.408491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.408456    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.408753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.408735    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6q7\" (UniqueName: \"kubernetes.io/projected/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-kube-api-access-8d6q7\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.409279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.409247    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.411862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.411836    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.417515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.417493    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:11.419772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.419750    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6q7\" (UniqueName: \"kubernetes.io/projected/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-kube-api-access-8d6q7\") pod \"maas-api-revoke-sub-f08dbefb-bgzkz\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.556540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.556467    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:11.850566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:11.847557    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"]
Nov 03 11:03:11.850566 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:03:11.849143    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7e6a4c_56ba_4f43_bfd5_70dab4dc814f.slice/crio-76fd185851a0974d5a1e01e630ba12095c5ae29815858b8220d6cf1bc89ef4c1 WatchSource:0}: Error finding container 76fd185851a0974d5a1e01e630ba12095c5ae29815858b8220d6cf1bc89ef4c1: Status 404 returned error can't find the container with id 76fd185851a0974d5a1e01e630ba12095c5ae29815858b8220d6cf1bc89ef4c1
Nov 03 11:03:12.091519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:12.091449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:12.553597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:12.553358    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz" event={"ID":"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f","Type":"ContainerStarted","Data":"49ef83e27c5762f4e21eb1d6391bc63d16f1a3ed786278d6f2f8c504f87f87fa"}
Nov 03 11:03:12.553597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:12.553427    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz" event={"ID":"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f","Type":"ContainerStarted","Data":"76fd185851a0974d5a1e01e630ba12095c5ae29815858b8220d6cf1bc89ef4c1"}
Nov 03 11:03:12.645634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:12.643798    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz" podStartSLOduration=1.643779364 podStartE2EDuration="1.643779364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:12.607177789 +0000 UTC m=+2022.812759842" watchObservedRunningTime="2026-11-03 11:03:12.643779364 +0000 UTC m=+2022.849361416"
Nov 03 11:03:12.891479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:12.891332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:13.578615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:13.573561    5817 generic.go:356] "Generic (PLEG): container finished" podID="de7e6a4c-56ba-4f43-bfd5-70dab4dc814f" containerID="49ef83e27c5762f4e21eb1d6391bc63d16f1a3ed786278d6f2f8c504f87f87fa" exitCode=0
Nov 03 11:03:13.578615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:13.573847    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz" event={"ID":"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f","Type":"ContainerDied","Data":"49ef83e27c5762f4e21eb1d6391bc63d16f1a3ed786278d6f2f8c504f87f87fa"}
Nov 03 11:03:13.805974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:13.805944    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:14.558514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:14.558461    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:14.863308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:14.863279    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:15.051906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.051871    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-kube-api-access-8d6q7\" (UniqueName: \"kubernetes.io/projected/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-kube-api-access-8d6q7\") pod \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") "
Nov 03 11:03:15.052111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.051926    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-xks-ca\") pod \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") "
Nov 03 11:03:15.052191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.052155    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-openshift-ca\") pod \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\" (UID: \"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f\") "
Nov 03 11:03:15.052609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.052538    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-openshift-ca" pod "de7e6a4c-56ba-4f43-bfd5-70dab4dc814f" (UID: "de7e6a4c-56ba-4f43-bfd5-70dab4dc814f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:03:15.052871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.052763    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:15.054179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.054148    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-xks-ca" pod "de7e6a4c-56ba-4f43-bfd5-70dab4dc814f" (UID: "de7e6a4c-56ba-4f43-bfd5-70dab4dc814f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:03:15.054306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.054287    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-kube-api-access-8d6q7" pod "de7e6a4c-56ba-4f43-bfd5-70dab4dc814f" (UID: "de7e6a4c-56ba-4f43-bfd5-70dab4dc814f"). InnerVolumeSpecName "kube-api-access-8d6q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:03:15.154432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.154315    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d6q7\" (UniqueName: \"kubernetes.io/projected/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-kube-api-access-8d6q7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:15.154432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.154363    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:15.232599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.231599    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:15.601601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.598673    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"
Nov 03 11:03:15.605596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.601881    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz" event={"ID":"de7e6a4c-56ba-4f43-bfd5-70dab4dc814f","Type":"ContainerDied","Data":"76fd185851a0974d5a1e01e630ba12095c5ae29815858b8220d6cf1bc89ef4c1"}
Nov 03 11:03:15.605596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.601955    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76fd185851a0974d5a1e01e630ba12095c5ae29815858b8220d6cf1bc89ef4c1"
Nov 03 11:03:15.947701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:15.940284    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:16.650344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:16.650306    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:16.815600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:16.814402    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:17.378345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:17.378309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:18.026601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:18.024089    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:18.173509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:18.173358    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:18.614277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:18.614237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:18.768487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:18.768451    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:19.026967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:19.026929    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:19.994109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:19.994074    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:20.797538 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:20.797503    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:21.696016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:21.695985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:22.423102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:22.423069    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:22.466300 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:03:22.466259    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:03:23.168143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:23.168104    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:23.348230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:23.348187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:23.865596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:23.859025    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:24.578937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:24.578822    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:24.785278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:24.785244    5817 generic.go:356] "Generic (PLEG): container finished" podID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerID="878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e" exitCode=0
Nov 03 11:03:24.785463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:24.785314    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" event={"ID":"129748c3-369d-4a33-9cda-7f5ebabf7d7e","Type":"ContainerDied","Data":"878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e"}
Nov 03 11:03:24.785801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:24.785785    5817 scope.go:122] "RemoveContainer" containerID="878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e"
Nov 03 11:03:24.787650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:24.787620    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:24.787754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:24.787688    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:24.787754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:24.787721    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:25.477160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:25.470213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:25.829621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:25.829515    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" event={"ID":"129748c3-369d-4a33-9cda-7f5ebabf7d7e","Type":"ContainerStarted","Data":"59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c"}
Nov 03 11:03:26.035414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:26.035373    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:03:26.037239 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:26.037207    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 11:03:26.037406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:26.037265    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 11:03:26.519389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:26.519306    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:27.036207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:27.036165    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 11:03:27.036742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:27.036223    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 11:03:27.190598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:27.190486    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:27.950223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:27.950183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:28.035377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:28.035329    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 11:03:28.035610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:28.035405    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 11:03:28.323476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:28.323392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:28.810599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:28.810205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:28.958595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:28.958518    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:29.036252 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.036209    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 11:03:29.036478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.036292    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 11:03:29.334596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.334002    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"]
Nov 03 11:03:29.335287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.335206    5817 cpu_manager.go:414] "containerMap: removing container" podUID="de7e6a4c-56ba-4f43-bfd5-70dab4dc814f" containerName="revoke-keys"
Nov 03 11:03:29.335287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.335233    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7e6a4c-56ba-4f43-bfd5-70dab4dc814f" containerName="revoke-keys"
Nov 03 11:03:29.335482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.335432    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7e6a4c-56ba-4f43-bfd5-70dab4dc814f" containerName="revoke-keys"
Nov 03 11:03:29.342547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.341477    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.342547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.341554    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"]
Nov 03 11:03:29.349616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.348339    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c8cb1f-openshift-default-dockercfg-8svfj\""
Nov 03 11:03:29.420722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.420676    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-credential-socket\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.421085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.421063    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-data\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.421300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.421285    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-certs\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.421516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.421500    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-token\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.422923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.422898    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-podinfo\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.423114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.423090    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:29.423209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.423106    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-socket\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.423314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.423301    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istiod-ca-cert\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.423428 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.423413    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-kube-api-access-q4qfl\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.423630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.423612    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-envoy\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.524968    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-token\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525020    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-podinfo\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525076    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-socket\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525103    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istiod-ca-cert\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525157    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-kube-api-access-q4qfl\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525225    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-envoy\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525425 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525403    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-credential-socket\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525505    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-data\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525541    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-certs\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.525972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525942    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-credential-socket\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.526103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.525976    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-certs\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.526103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.526023    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-data\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.526230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.526214    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istiod-ca-cert\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.526524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.526503    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-socket\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.527984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.527952    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-envoy\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.528073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.528018    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-podinfo\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.534487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.534442    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-token\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.535838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.535798    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-kube-api-access-q4qfl\") pod \"e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") " pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.607780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.607523    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:29.665653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.658913    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:29.785625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:29.773687    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:03:30.036182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.035524    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 11:03:30.036182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.035725    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 11:03:30.185014 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:03:30.184986    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc7ee5d_3359_4fb2_b5d4_31123a6e54ac.slice/crio-834b9be1a5ce1dc98b24d1dddfaed74be662909925ee912fd17e14ee6cee1575 WatchSource:0}: Error finding container 834b9be1a5ce1dc98b24d1dddfaed74be662909925ee912fd17e14ee6cee1575: Status 404 returned error can't find the container with id 834b9be1a5ce1dc98b24d1dddfaed74be662909925ee912fd17e14ee6cee1575
Nov 03 11:03:30.185404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.185382    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"]
Nov 03 11:03:30.189986 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.189131    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:30.189986 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.189220    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:30.189986 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.189265    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:30.386646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.381972    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:30.910598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.907691    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" event={"ID":"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac","Type":"ContainerStarted","Data":"7e13ffc7e849e60b43304880b12032ab6722d710256ec111a7136caf7d268848"}
Nov 03 11:03:30.910598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:30.907761    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" event={"ID":"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac","Type":"ContainerStarted","Data":"834b9be1a5ce1dc98b24d1dddfaed74be662909925ee912fd17e14ee6cee1575"}
Nov 03 11:03:31.036045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:31.036005    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 11:03:31.036243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:31.036071    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 11:03:31.339602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:31.338624    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:31.662805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:31.662714    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:31.669357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:31.669332    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:31.696191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:31.696139    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" podStartSLOduration=2.696122033 podStartE2EDuration="2.696122033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:30.941611227 +0000 UTC m=+2041.147193269" watchObservedRunningTime="2026-11-03 11:03:31.696122033 +0000 UTC m=+2041.901704084"
Nov 03 11:03:31.917321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:31.917247    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:31.918247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:31.918213    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:03:32.041044 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:32.041018    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:03:32.042177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:32.042158    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:03:32.334598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:32.334066    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:33.171939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:33.171907    5817 scope.go:122] "RemoveContainer" containerID="88cc6413f8ffc9228bb89c9a57c0f0b146637ddf1d609ebe20dc7a13505fd4f4"
Nov 03 11:03:33.183131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:33.183111    5817 scope.go:122] "RemoveContainer" containerID="1bdfd8232f9ff680032a4748aa977a956eb58b570b2e284062e2c89117b6a8f3"
Nov 03 11:03:33.193767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:33.193746    5817 scope.go:122] "RemoveContainer" containerID="e08b0a2dbbc97fd38c54a2880d03c2c0b15ea3719e26f062c209a79b98ee1c39"
Nov 03 11:03:34.159674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.159643    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"]
Nov 03 11:03:34.164387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.164362    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.175284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.175259    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"]
Nov 03 11:03:34.222328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.222294    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d9f2ce\""
Nov 03 11:03:34.222640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.222616    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d9f2ce-dockercfg-6ntgt\""
Nov 03 11:03:34.229276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.229246    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjghv\" (UniqueName: \"kubernetes.io/projected/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-kube-api-access-vjghv\") pod \"payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f\" (UID: \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.229681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.229585    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f\" (UID: \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.229681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.229632    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:34.341775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.341724    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjghv\" (UniqueName: \"kubernetes.io/projected/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-kube-api-access-vjghv\") pod \"payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f\" (UID: \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.366594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.343927    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f\" (UID: \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.366594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.353992    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f\" (UID: \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.372593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.366958    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjghv\" (UniqueName: \"kubernetes.io/projected/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-kube-api-access-vjghv\") pod \"payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f\" (UID: \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.484600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.484135    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.826615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.826583    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"]
Nov 03 11:03:34.828060 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:03:34.828022    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec2a5fd_6fc8_4f67_982f_c2cc457bc645.slice/crio-8a767246a9ab6643c3d134c6f9e64df4cfbbb455e13ade5dd85f8fd769843884 WatchSource:0}: Error finding container 8a767246a9ab6643c3d134c6f9e64df4cfbbb455e13ade5dd85f8fd769843884: Status 404 returned error can't find the container with id 8a767246a9ab6643c3d134c6f9e64df4cfbbb455e13ade5dd85f8fd769843884
Nov 03 11:03:34.960926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.960887    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f" event={"ID":"aec2a5fd-6fc8-4f67-982f-c2cc457bc645","Type":"ContainerStarted","Data":"8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab"}
Nov 03 11:03:34.961100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.960948    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f" event={"ID":"aec2a5fd-6fc8-4f67-982f-c2cc457bc645","Type":"ContainerStarted","Data":"8a767246a9ab6643c3d134c6f9e64df4cfbbb455e13ade5dd85f8fd769843884"}
Nov 03 11:03:34.961100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.961010    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:34.985444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:34.985388    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f" podStartSLOduration=0.985374287 podStartE2EDuration="985.374287ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:34.981938659 +0000 UTC m=+2045.187520711" watchObservedRunningTime="2026-11-03 11:03:34.985374287 +0000 UTC m=+2045.190956339"
Nov 03 11:03:35.206073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:35.206038    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:03:35.208713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:35.207917    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:35.976610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:35.976419    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:37.154232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:37.149918    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:37.996595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:37.996112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:38.714592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:38.709187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:39.026842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:39.026755    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:39.205129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:39.205083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:39.385223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:39.385137    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:39.580800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:39.580764    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:39.782704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:39.782656    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:40.085658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:40.082653    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:40.327351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:40.327315    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:40.513812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:40.513771    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:40.794651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:40.788790    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:41.076135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:41.076051    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:41.273768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:41.273732    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:41.450706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:41.450672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:41.665266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:41.665236    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:41.946602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:41.946550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:42.895645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:42.895613    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:43.193412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:43.193375    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:43.578561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:43.578483    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:43.793481 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:43.793451    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:44.159842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:44.159807    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:44.888420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:44.888385    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:45.098472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:45.098436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:45.606050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:45.605975    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:45.978157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:45.978132    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:03:46.684185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:46.684148    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:47.629598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:47.625814    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:48.396296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:48.389191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:49.115851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:49.115809    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:49.782005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:49.781967    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:50.407398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:50.407315    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:51.092439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:51.092407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:51.777653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:51.775248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:52.413821 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:03:52.413786    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:03:52.475438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:52.475406    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:53.116246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:53.116206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:53.771853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:53.771816    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:53.932854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:53.932800    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:54.663609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:54.663562    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"]
Nov 03 11:03:54.734513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:54.734481    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:55.569651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:55.569605    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:56.304029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:56.303988    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:56.893953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:56.893671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:57.679185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:57.679135    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:58.485877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:58.485188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:59.134184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:59.134142    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:03:59.664179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:59.664141    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q" podUID="dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" containerName="payload-processing" containerID="cri-o://b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008" gracePeriod=25
Nov 03 11:03:59.791600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:03:59.783746    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:00.028757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.028721    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:04:00.077320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.077281    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-provider-credentials\") pod \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") "
Nov 03 11:04:00.077500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.077388    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-routing-overlay-volume\") pod \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") "
Nov 03 11:04:00.077500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.077473    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-kube-api-access-w2x2g\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-kube-api-access-w2x2g\") pod \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") "
Nov 03 11:04:00.077826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.077566    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-plugins-config-volume\") pod \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\" (UID: \"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f\") "
Nov 03 11:04:00.078685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.078659    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-plugins-config-volume" pod "dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" (UID: "dbd0f19a-dc47-4894-a20b-8912e2a6ca0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:00.079193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.079170    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-routing-overlay-volume" pod "dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" (UID: "dbd0f19a-dc47-4894-a20b-8912e2a6ca0f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:00.081430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.081402    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-provider-credentials" pod "dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" (UID: "dbd0f19a-dc47-4894-a20b-8912e2a6ca0f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:00.084744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.084718    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-kube-api-access-w2x2g" pod "dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" (UID: "dbd0f19a-dc47-4894-a20b-8912e2a6ca0f"). InnerVolumeSpecName "kube-api-access-w2x2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:00.179294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.179261    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:00.179294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.179290    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:00.179294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.179301    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2x2g\" (UniqueName: \"kubernetes.io/projected/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-kube-api-access-w2x2g\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:00.179792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.179310    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:00.411998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.411892    5817 generic.go:356] "Generic (PLEG): container finished" podID="dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" containerID="b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008" exitCode=0
Nov 03 11:04:00.411998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.411951    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"
Nov 03 11:04:00.411998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.411982    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q" event={"ID":"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f","Type":"ContainerDied","Data":"b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008"}
Nov 03 11:04:00.412316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.412043    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q" event={"ID":"dbd0f19a-dc47-4894-a20b-8912e2a6ca0f","Type":"ContainerDied","Data":"a0789fde0572d8aed4a4d36b93c3659613217fcac7d562e182b3f709d0b86034"}
Nov 03 11:04:00.412316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.412072    5817 scope.go:122] "RemoveContainer" containerID="b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008"
Nov 03 11:04:00.424228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.424212    5817 scope.go:122] "RemoveContainer" containerID="b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008"
Nov 03 11:04:00.424511 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:04:00.424492    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008\": container with ID starting with b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008 not found: ID does not exist" containerID="b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008"
Nov 03 11:04:00.424599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.424517    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008"} err="failed to get container status \"b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008\": rpc error: code = NotFound desc = could not find container \"b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008\": container with ID starting with b86deff7628d6102ca4b1715eb673461d4ce68e1e9fbd533fc499729528aa008 not found: ID does not exist"
Nov 03 11:04:00.443382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.443361    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"]
Nov 03 11:04:00.447684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.447665    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dldw4q"]
Nov 03 11:04:00.477987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:00.477952    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:01.126292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:01.126196    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:01.787095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:01.787061    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:01.965403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:01.961379    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:01.970276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:01.970241    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" path="/var/lib/kubelet/pods/dbd0f19a-dc47-4894-a20b-8912e2a6ca0f/volumes"
Nov 03 11:04:02.679595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:02.677645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:03.348953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:03.348917    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:03.947890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:03.947853    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:04.591505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:04.589207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:05.279598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:05.279494    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:05.895240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:05.892068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:06.591643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:06.582311    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:07.208600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:07.203509    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:07.653098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:07.653018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:08.395593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:08.387718    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:09.683652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:09.672165    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:10.569556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:10.569522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:11.266855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.266819    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:11.435611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.435271    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:11.701733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.701702    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"]
Nov 03 11:04:11.702727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.702693    5817 cpu_manager.go:414] "containerMap: removing container" podUID="dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" containerName="payload-processing"
Nov 03 11:04:11.702727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.702717    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" containerName="payload-processing"
Nov 03 11:04:11.702921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.702883    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd0f19a-dc47-4894-a20b-8912e2a6ca0f" containerName="payload-processing"
Nov 03 11:04:11.708952 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.708932    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.717634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.717589    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"]
Nov 03 11:04:11.719433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.719411    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"routing-overlay\""
Nov 03 11:04:11.719628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.719592    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-e2e--b8bff1be-dockercfg-45kcv\""
Nov 03 11:04:11.719722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.719428    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-586b7631a686\""
Nov 03 11:04:11.719722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.719465    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-e8a75993ff8f\""
Nov 03 11:04:11.719722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.719526    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"kube-root-ca.crt\""
Nov 03 11:04:11.719722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.719654    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"openshift-service-ca.crt\""
Nov 03 11:04:11.719722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.719417    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fb5766\""
Nov 03 11:04:11.719722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.719700    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-e40ddfc7cf41\""
Nov 03 11:04:11.835707 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.835666    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.835894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.835734    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.835894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.835806    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.835894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.835886    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnn8t\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-kube-api-access-gnn8t\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.937922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.937881    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.938125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.937956    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.938125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.938033    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.938251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.938140    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnn8t\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-kube-api-access-gnn8t\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.938842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.938819    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.938842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.938835    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.941079 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.941061    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:11.946075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:11.946049    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnn8t\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-kube-api-access-gnn8t\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:12.044534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.044462    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:12.302134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.302041    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:12.401134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.398667    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"]
Nov 03 11:04:12.442864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.442827    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:12.752598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.748769    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"]
Nov 03 11:04:12.763983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.759248    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.764798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.764775    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"agc-cred-ff8b9f4705b7\""
Nov 03 11:04:12.764999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.764975    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"payload-processing-external-model-e2e--0d6b7192-dockercfg-97m2g\""
Nov 03 11:04:12.765101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.765090    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"openshift-service-ca.crt\""
Nov 03 11:04:12.765168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.765155    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b76c62\""
Nov 03 11:04:12.765262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.765246    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"routing-overlay\""
Nov 03 11:04:12.765363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.764920    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"kube-root-ca.crt\""
Nov 03 11:04:12.765518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.765494    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"agc-cred-bb5be166ad5d\""
Nov 03 11:04:12.783919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.783892    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v" event={"ID":"b642cdd7-025a-460e-907e-2927365d6cb7","Type":"ContainerStarted","Data":"fbe1c51232e0b737e2e511002e756f51e5dd183056a6b3d6b32fe3e7670104fe"}
Nov 03 11:04:12.791971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.791945    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"]
Nov 03 11:04:12.792124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.792004    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v" event={"ID":"b642cdd7-025a-460e-907e-2927365d6cb7","Type":"ContainerStarted","Data":"eae5c5be18de37144828c67b4a0f14d7d294e62abf896fb63bbef8b6685011b7"}
Nov 03 11:04:12.794047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.794028    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:12.815076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.815027    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v" podStartSLOduration=1.8150108230000002 podStartE2EDuration="1.815010823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:12.811330374 +0000 UTC m=+2083.016912428" watchObservedRunningTime="2026-11-03 11:04:12.815010823 +0000 UTC m=+2083.020592875"
Nov 03 11:04:12.856686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.856661    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7xq\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-kube-api-access-dq7xq\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.856796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.856720    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.856940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.856922    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.856980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.856956    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.958918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.958882    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.959178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.959145    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.959317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.959203    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.959416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.959391    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7xq\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-kube-api-access-dq7xq\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.959828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.959806    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.959949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.959888    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.961851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.961832    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:12.967261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:12.967242    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7xq\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-kube-api-access-dq7xq\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:13.094215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:13.094127    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:13.382721 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:04:13.382692    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76955372_6a2e_439f_9aa8_34f0603ca75a.slice/crio-bd9c8d84c3a01451660927723f26db727c9d8be851a4c8a49608cfc7822817d4 WatchSource:0}: Error finding container bd9c8d84c3a01451660927723f26db727c9d8be851a4c8a49608cfc7822817d4: Status 404 returned error can't find the container with id bd9c8d84c3a01451660927723f26db727c9d8be851a4c8a49608cfc7822817d4
Nov 03 11:04:13.383038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:13.382775    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"]
Nov 03 11:04:13.686133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:13.686100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:13.796141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:13.796103    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9" event={"ID":"76955372-6a2e-439f-9aa8-34f0603ca75a","Type":"ContainerStarted","Data":"047690af2794bfb195888cdd10148ad31caefe9d1803fc836416fcdfebd2acfb"}
Nov 03 11:04:13.796346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:13.796165    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9" event={"ID":"76955372-6a2e-439f-9aa8-34f0603ca75a","Type":"ContainerStarted","Data":"bd9c8d84c3a01451660927723f26db727c9d8be851a4c8a49608cfc7822817d4"}
Nov 03 11:04:13.796346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:13.796215    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:13.817324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:13.817272    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9" podStartSLOduration=1.817258735 podStartE2EDuration="1.817258735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:13.815009223 +0000 UTC m=+2084.020591276" watchObservedRunningTime="2026-11-03 11:04:13.817258735 +0000 UTC m=+2084.022840785"
Nov 03 11:04:14.769599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:14.765018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:15.446147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:15.446108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:16.175599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:16.170596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:16.890598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:16.888108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:17.561610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:17.560968    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:17.860037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:17.859944    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:18.272389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.272349    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:18.598047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.597960    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"]
Nov 03 11:04:18.604059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.604033    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.610124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.609400    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d9f2ce-openshift-default-dockercfg-zldhk\""
Nov 03 11:04:18.613218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.613181    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"]
Nov 03 11:04:18.669924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.667431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:18.766190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766157    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.766366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766214    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.766366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766268    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.766366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766322    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.766479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766432    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5lpn\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-kube-api-access-m5lpn\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.766479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766466    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.766603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766564    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.766644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766634    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.766689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.766666    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868176    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868233    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868263    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868311    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868394    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5lpn\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-kube-api-access-m5lpn\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868426    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868511    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868594    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868638    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.868839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.868796    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.869029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.869002    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.869098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.869056    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.869164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.869144    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.869211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.869167    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.870656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.870636    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.870929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.870912    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.876907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.876877    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.877039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.876884    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5lpn\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-kube-api-access-m5lpn\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:18.938849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:18.938824    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:19.008598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.008376    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:19.310852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.310657    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"]
Nov 03 11:04:19.312099 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:04:19.312071    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aecee12_bbb1_47ff_86fc_dd034ed4ca05.slice/crio-09374aa6220b16bb566557c2a4d4b5ffbf590687a2f85bc30834cd2b726d89d6 WatchSource:0}: Error finding container 09374aa6220b16bb566557c2a4d4b5ffbf590687a2f85bc30834cd2b726d89d6: Status 404 returned error can't find the container with id 09374aa6220b16bb566557c2a4d4b5ffbf590687a2f85bc30834cd2b726d89d6
Nov 03 11:04:19.316203 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.316164    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:19.316318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.316249    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:19.316318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.316292    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:19.751599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.751091    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:19.861067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.861035    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" event={"ID":"9aecee12-bbb1-47ff-86fc-dd034ed4ca05","Type":"ContainerStarted","Data":"93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627"}
Nov 03 11:04:19.861242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.861097    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" event={"ID":"9aecee12-bbb1-47ff-86fc-dd034ed4ca05","Type":"ContainerStarted","Data":"09374aa6220b16bb566557c2a4d4b5ffbf590687a2f85bc30834cd2b726d89d6"}
Nov 03 11:04:19.890075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.890020    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" podStartSLOduration=1.890002642 podStartE2EDuration="1.890002642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:19.888052392 +0000 UTC m=+2090.093634443" watchObservedRunningTime="2026-11-03 11:04:19.890002642 +0000 UTC m=+2090.095584694"
Nov 03 11:04:19.915540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.915502    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:19.939662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:19.939635    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:20.051206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:20.051123    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:20.219813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:20.219780    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:20.527464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:20.527432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:20.681330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:20.681285    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:20.830839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:20.830724    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:20.949595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:20.945760    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 03 11:04:20.949595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:20.945823    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 03 11:04:21.353779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:21.353722    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:21.527226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:21.527194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:21.855270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:21.855239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:21.939870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:21.939836    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 03 11:04:21.940090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:21.939892    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 03 11:04:22.452121 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:04:22.452086    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:04:22.939740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:22.939706    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 03 11:04:22.940261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:22.939775    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 03 11:04:23.079721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:23.079682    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:23.519484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:23.519452    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:23.661788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:23.661750    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:23.940171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:23.940136    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 03 11:04:23.940589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:23.940198    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 03 11:04:24.167261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:24.167226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:24.318685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:24.318598    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:24.816503 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:24.816459    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:04:24.817933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:24.817911    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:04:24.940218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:24.940184    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 03 11:04:24.940837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:24.940807    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 03 11:04:25.560013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:25.559967    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:25.939287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:25.939257    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 03 11:04:25.939436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:25.939326    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 03 11:04:26.528337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:26.528301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:26.944421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:26.944394    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:26.944870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:26.944850    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:26.945671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:26.945650    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:04:27.389654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:27.380194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:28.078855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:28.078817    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:28.675358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:28.675321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:28.971334 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:28.971285    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:29.448671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:29.448639    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:30.082609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:30.078872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:30.364252 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:30.364154    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:30.493655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:30.493617    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:30.634430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:30.634337    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:30.814599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:30.801044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:31.197146 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:31.197107    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:31.883659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:31.880058    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:32.556344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:32.556308    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:33.181153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:33.181110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:33.245141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:33.245112    5817 scope.go:122] "RemoveContainer" containerID="335930e1c9b3e116dd56eeaad03f84353bd2564a38a3c86d584c126b5aeda36b"
Nov 03 11:04:33.872594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:33.869189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:34.582327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:34.582223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:35.279368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:35.279333    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:36.072420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:36.072386    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:36.219093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:36.219058    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:36.607339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:36.607242    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:37.363439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:37.363400    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:37.553593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:37.552095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:38.465644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:38.465608    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:39.283560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:39.283479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:39.963035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:39.962990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:40.734631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:40.733350    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:41.436589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:41.436537    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:42.074929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:42.074895    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:42.782559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:42.782432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:42.977166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:42.977126    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:43.133193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:43.133104    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:43.275257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:43.275214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:43.435021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:43.434987    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:43.738288 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:43.738199    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:44.008859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.008770    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:44.190862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.190824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:44.502517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.502485    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"]
Nov 03 11:04:44.510281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.510211    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.514388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.514362    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"]
Nov 03 11:04:44.515768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.515747    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"agc-cred-5d8152330398\""
Nov 03 11:04:44.515903 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.515783    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"agc-cred-687d7e71d6a7\""
Nov 03 11:04:44.517870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.517847    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"payload-processing-external-model-e2e--885106fd-dockercfg-w2xvw\""
Nov 03 11:04:44.518039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.517876    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"routing-overlay\""
Nov 03 11:04:44.518039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.517922    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"openshift-service-ca.crt\""
Nov 03 11:04:44.518233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.517991    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-56687c\""
Nov 03 11:04:44.518233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.517997    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"kube-root-ca.crt\""
Nov 03 11:04:44.608348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.608314    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.608633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.608356    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtvn\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-kube-api-access-gxtvn\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.608765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.608737    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.608864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.608847    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.710346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.710312    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.710533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.710433    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.710533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.710526    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.710653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.710552    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtvn\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-kube-api-access-gxtvn\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.711661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.711623    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.711802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.711723    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.713494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.713473    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.720164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.720142    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtvn\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-kube-api-access-gxtvn\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:44.830342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:44.830258    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:45.026596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:45.026149    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:45.132205 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:04:45.132170    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198ace54_2e55_4014_92c8_3b409bae9f96.slice/crio-14edeaa3cea193b02f67ffb7d1f3ff5ff776672615e9710f54f5ecd21b65503d WatchSource:0}: Error finding container 14edeaa3cea193b02f67ffb7d1f3ff5ff776672615e9710f54f5ecd21b65503d: Status 404 returned error can't find the container with id 14edeaa3cea193b02f67ffb7d1f3ff5ff776672615e9710f54f5ecd21b65503d
Nov 03 11:04:45.133209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:45.133149    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"]
Nov 03 11:04:45.252709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:45.252679    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x" event={"ID":"198ace54-2e55-4014-92c8-3b409bae9f96","Type":"ContainerStarted","Data":"43bc921b58a71b24886981b5f3c44cb61b47dc35c77bb13be8d270b3a83b53cf"}
Nov 03 11:04:45.252890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:45.252729    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x" event={"ID":"198ace54-2e55-4014-92c8-3b409bae9f96","Type":"ContainerStarted","Data":"14edeaa3cea193b02f67ffb7d1f3ff5ff776672615e9710f54f5ecd21b65503d"}
Nov 03 11:04:45.252890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:45.252771    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:45.271111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:45.271059    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x" podStartSLOduration=1.27104593 podStartE2EDuration="1.27104593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:45.269972038 +0000 UTC m=+2115.475554088" watchObservedRunningTime="2026-11-03 11:04:45.27104593 +0000 UTC m=+2115.476628025"
Nov 03 11:04:45.314165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:45.314135    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:04:46.017597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:46.016614    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:46.454479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:46.454443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:47.174596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.163066    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"]
Nov 03 11:04:47.223881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.223849    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.247601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.247223    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:04:47.251592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.249299    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:04:47.309722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.306291    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"]
Nov 03 11:04:47.397757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.397718    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.398286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.398259    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frg98\" (UniqueName: \"kubernetes.io/projected/f7d36e5e-b748-484f-b41c-c3a5c91e6172-kube-api-access-frg98\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.398496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.398475    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.579595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.575699    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg98\" (UniqueName: \"kubernetes.io/projected/f7d36e5e-b748-484f-b41c-c3a5c91e6172-kube-api-access-frg98\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.579595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.575781    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.579595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.576213    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.589344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.589310    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.603496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.603466    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.627641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.625791    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frg98\" (UniqueName: \"kubernetes.io/projected/f7d36e5e-b748-484f-b41c-c3a5c91e6172-kube-api-access-frg98\") pod \"maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.649420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.649385    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:47.968324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:47.968295    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"]
Nov 03 11:04:47.971040 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:04:47.971006    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d36e5e_b748_484f_b41c_c3a5c91e6172.slice/crio-96b268f67222b2f190ff73dc8e88f4946251fee2e3dbf50766ac2f9829459580 WatchSource:0}: Error finding container 96b268f67222b2f190ff73dc8e88f4946251fee2e3dbf50766ac2f9829459580: Status 404 returned error can't find the container with id 96b268f67222b2f190ff73dc8e88f4946251fee2e3dbf50766ac2f9829459580
Nov 03 11:04:48.143534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:48.143500    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:48.325692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:48.325616    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr" event={"ID":"f7d36e5e-b748-484f-b41c-c3a5c91e6172","Type":"ContainerStarted","Data":"96b268f67222b2f190ff73dc8e88f4946251fee2e3dbf50766ac2f9829459580"}
Nov 03 11:04:48.803765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:48.802007    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:49.336069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:49.336031    5817 generic.go:356] "Generic (PLEG): container finished" podID="f7d36e5e-b748-484f-b41c-c3a5c91e6172" containerID="d5e3efe8c1e91cfbefbd614c85201bbbca31a460f0ce9e55c08c7eb602d02e42" exitCode=0
Nov 03 11:04:49.336244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:49.336120    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr" event={"ID":"f7d36e5e-b748-484f-b41c-c3a5c91e6172","Type":"ContainerDied","Data":"d5e3efe8c1e91cfbefbd614c85201bbbca31a460f0ce9e55c08c7eb602d02e42"}
Nov 03 11:04:49.576445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:49.576361    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:49.747779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:49.747742    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:49.881279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:49.881189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:50.482773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:50.468015    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:51.008670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.008632    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"]
Nov 03 11:04:51.043607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.043379    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.047630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.047104    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"agc-cred-307180d89aed\""
Nov 03 11:04:51.101777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.101743    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"]
Nov 03 11:04:51.102785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.102746    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxr5\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-kube-api-access-jdxr5\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.102928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.102886    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.105182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.103189    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.105182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.103374    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.226606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.214595    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.226606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.214824    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.226606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.214932    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.226606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.214995    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxr5\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-kube-api-access-jdxr5\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.226606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.216040    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.226606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.216704    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.246377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.244420    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxr5\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-kube-api-access-jdxr5\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.248177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.248152    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.359641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.359617    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:51.466525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.466466    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:51.472333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.472301    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:51.542359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.542325    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-openshift-ca\") pod \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") "
Nov 03 11:04:51.542840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.542818    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-xks-ca\") pod \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") "
Nov 03 11:04:51.543153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.543131    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d36e5e-b748-484f-b41c-c3a5c91e6172-kube-api-access-frg98\" (UniqueName: \"kubernetes.io/projected/f7d36e5e-b748-484f-b41c-c3a5c91e6172-kube-api-access-frg98\") pod \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\" (UID: \"f7d36e5e-b748-484f-b41c-c3a5c91e6172\") "
Nov 03 11:04:51.543371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.543197    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-openshift-ca" pod "f7d36e5e-b748-484f-b41c-c3a5c91e6172" (UID: "f7d36e5e-b748-484f-b41c-c3a5c91e6172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:51.544338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.544290    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:51.546787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.546511    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d36e5e-b748-484f-b41c-c3a5c91e6172-kube-api-access-frg98" pod "f7d36e5e-b748-484f-b41c-c3a5c91e6172" (UID: "f7d36e5e-b748-484f-b41c-c3a5c91e6172"). InnerVolumeSpecName "kube-api-access-frg98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:51.547948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.547808    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-xks-ca" pod "f7d36e5e-b748-484f-b41c-c3a5c91e6172" (UID: "f7d36e5e-b748-484f-b41c-c3a5c91e6172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:04:51.646241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.646209    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frg98\" (UniqueName: \"kubernetes.io/projected/f7d36e5e-b748-484f-b41c-c3a5c91e6172-kube-api-access-frg98\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:51.646241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.646245    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d36e5e-b748-484f-b41c-c3a5c91e6172-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:51.679065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.679028    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr" event={"ID":"f7d36e5e-b748-484f-b41c-c3a5c91e6172","Type":"ContainerDied","Data":"96b268f67222b2f190ff73dc8e88f4946251fee2e3dbf50766ac2f9829459580"}
Nov 03 11:04:51.679211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.679073    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"
Nov 03 11:04:51.679211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.679124    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96b268f67222b2f190ff73dc8e88f4946251fee2e3dbf50766ac2f9829459580"
Nov 03 11:04:51.857820 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:04:51.857789    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16535339_e77f_4fd4_8554_447e2e4abc8e.slice/crio-4590333e9e19c294d5408d8feebc40d6294aef9fac0f8e7663ff55c857bbcae6 WatchSource:0}: Error finding container 4590333e9e19c294d5408d8feebc40d6294aef9fac0f8e7663ff55c857bbcae6: Status 404 returned error can't find the container with id 4590333e9e19c294d5408d8feebc40d6294aef9fac0f8e7663ff55c857bbcae6
Nov 03 11:04:51.859554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:51.859522    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"]
Nov 03 11:04:52.309600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:52.309212    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:52.464440 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:04:52.464404    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:04:52.723600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:52.723431    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw" event={"ID":"16535339-e77f-4fd4-8554-447e2e4abc8e","Type":"ContainerStarted","Data":"3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc"}
Nov 03 11:04:52.723600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:52.723494    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw" event={"ID":"16535339-e77f-4fd4-8554-447e2e4abc8e","Type":"ContainerStarted","Data":"4590333e9e19c294d5408d8feebc40d6294aef9fac0f8e7663ff55c857bbcae6"}
Nov 03 11:04:52.723875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:52.723695    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:04:52.743819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:52.743772    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw" podStartSLOduration=2.743754001 podStartE2EDuration="2.743754001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:52.742022506 +0000 UTC m=+2122.947604556" watchObservedRunningTime="2026-11-03 11:04:52.743754001 +0000 UTC m=+2122.949336058"
Nov 03 11:04:53.060654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:53.057307    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:53.852057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:53.852013    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:54.567771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:54.567676    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:55.491379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:55.491338    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:55.849255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:55.849089    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:56.264304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:56.264280    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:04:56.543310 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:56.543223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:57.164718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:57.164681    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:57.317223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:57.317174    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:57.462482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:57.462451    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:58.348332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:58.348290    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:58.797374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:58.797345    5817 generic.go:356] "Generic (PLEG): container finished" podID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerID="7e13ffc7e849e60b43304880b12032ab6722d710256ec111a7136caf7d268848" exitCode=0
Nov 03 11:04:58.797633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:58.797413    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" event={"ID":"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac","Type":"ContainerDied","Data":"7e13ffc7e849e60b43304880b12032ab6722d710256ec111a7136caf7d268848"}
Nov 03 11:04:58.797850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:58.797835    5817 scope.go:122] "RemoveContainer" containerID="7e13ffc7e849e60b43304880b12032ab6722d710256ec111a7136caf7d268848"
Nov 03 11:04:58.799586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:58.799533    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:58.799715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:58.799644    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:58.799715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:58.799695    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:58.966602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:58.957964    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:59.650391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:59.650298    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:04:59.841600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:04:59.836984    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" event={"ID":"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac","Type":"ContainerStarted","Data":"4184ea323bcb87cbc887da1289b776174d78df2f869c2223127bbacb23201f81"}
Nov 03 11:05:00.461518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:00.450255    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:00.662642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:00.662611    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:05:00.663798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:00.663773    5817 patch_prober.go:29] interesting pod/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 11:05:00.663953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:00.663861    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 11:05:01.148679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:01.147272    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:01.662902 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:01.662868    5817 patch_prober.go:29] interesting pod/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 11:05:01.663325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:01.662915    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 11:05:01.774627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:01.774547    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:01.932967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:01.932941    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:05:02.609233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:02.609138    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:02.680606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:02.676981    5817 patch_prober.go:29] interesting pod/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 11:05:02.680606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:02.677042    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 11:05:02.753779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:02.751496    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:03.270600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.268750    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:03.648444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.648044    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"]
Nov 03 11:05:03.649682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.649663    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f7d36e5e-b748-484f-b41c-c3a5c91e6172" containerName="revoke-keys"
Nov 03 11:05:03.649682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.649682    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d36e5e-b748-484f-b41c-c3a5c91e6172" containerName="revoke-keys"
Nov 03 11:05:03.649866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.649826    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d36e5e-b748-484f-b41c-c3a5c91e6172" containerName="revoke-keys"
Nov 03 11:05:03.653983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.653966    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.664199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.664174    5817 patch_prober.go:29] interesting pod/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 11:05:03.664397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.664372    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 11:05:03.669670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.668076    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"]
Nov 03 11:05:03.685375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.685346    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"]
Nov 03 11:05:03.734830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.733300    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.743632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.737396    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"]
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752324    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752385    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752473    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752496    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxnr\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-kube-api-access-5hxnr\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752626    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752698    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752759    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752837    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.753513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.752896    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.760931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.760912    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:05:03.829299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.829256    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"]
Nov 03 11:05:03.857919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.857887    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.858216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.858194    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.858377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.858357    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxnr\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-kube-api-access-5hxnr\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.858564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.858546    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.859607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.858735    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.859857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.859801    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.860085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860064    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7948a568-6219-4907-a796-f32e9c6c9bac-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.860264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860204    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.860396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860375    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.860486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860457    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55zc\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-kube-api-access-t55zc\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.860548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860506    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.860694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860641    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.860694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860685    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.860808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860757    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.860867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860825    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7948a568-6219-4907-a796-f32e9c6c9bac-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.860867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860855    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.861036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.860949    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:03.861036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.861003    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.861149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.861060    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.861493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.861472    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.861947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.861879    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.862308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.862218    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.862609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.862565    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.862908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.862891    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.865597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.865410    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.867313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.867091    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxnr\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-kube-api-access-5hxnr\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:03.869938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.869636    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.968827    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.968902    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.968950    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7948a568-6219-4907-a796-f32e9c6c9bac-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.969030    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55zc\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-kube-api-access-t55zc\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.969116    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.969223    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7948a568-6219-4907-a796-f32e9c6c9bac-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.969248    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.969283    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.002671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:03.969410    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.036641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.003715    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.036641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.020019    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.036641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.021745    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7948a568-6219-4907-a796-f32e9c6c9bac-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.036641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.022856    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:04.036641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.035899    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55zc\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-kube-api-access-t55zc\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.037735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.037704    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.037899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.037876    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7948a568-6219-4907-a796-f32e9c6c9bac-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.038055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.038035    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.038700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.038676    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.041332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.040652    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:04.045621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.043455    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.065107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.065081    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:04.381377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.381302    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"]
Nov 03 11:05:04.385265 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:05:04.385230    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b519ba_f9f9_46a4_b91a_1dc194d80f6a.slice/crio-b94e4a521565a9379430e60b57ca8dd9aa9a04e51ac437047417cc27392550f7 WatchSource:0}: Error finding container b94e4a521565a9379430e60b57ca8dd9aa9a04e51ac437047417cc27392550f7: Status 404 returned error can't find the container with id b94e4a521565a9379430e60b57ca8dd9aa9a04e51ac437047417cc27392550f7
Nov 03 11:05:04.419277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.419038    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:04.419277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.419105    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:04.419277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.419133    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:04.421795 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:05:04.421767    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7948a568_6219_4907_a796_f32e9c6c9bac.slice/crio-bf37b05ee08335313d55081d78ced75a697a14eff07c77b01989418c822f04ec WatchSource:0}: Error finding container bf37b05ee08335313d55081d78ced75a697a14eff07c77b01989418c822f04ec: Status 404 returned error can't find the container with id bf37b05ee08335313d55081d78ced75a697a14eff07c77b01989418c822f04ec
Nov 03 11:05:04.424688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.424661    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"]
Nov 03 11:05:04.434983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.434944    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:04.435088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.435030    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:04.435088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.435075    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:04.661646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.659147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:04.678598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.672012    5817 patch_prober.go:29] interesting pod/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 11:05:04.678598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:04.672073    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 11:05:05.060304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.060263    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" event={"ID":"7948a568-6219-4907-a796-f32e9c6c9bac","Type":"ContainerStarted","Data":"b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03"}
Nov 03 11:05:05.060855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.060323    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" event={"ID":"7948a568-6219-4907-a796-f32e9c6c9bac","Type":"ContainerStarted","Data":"bf37b05ee08335313d55081d78ced75a697a14eff07c77b01989418c822f04ec"}
Nov 03 11:05:05.063313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.063275    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" event={"ID":"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a","Type":"ContainerStarted","Data":"c5b636f3ee37f1b07c8c7f55802f2fd9f9b7b524e8a85940db2c849b50f6c7df"}
Nov 03 11:05:05.063478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.063331    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" event={"ID":"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a","Type":"ContainerStarted","Data":"b94e4a521565a9379430e60b57ca8dd9aa9a04e51ac437047417cc27392550f7"}
Nov 03 11:05:05.065724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.065700    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:05.067421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.067394    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 11:05:05.067556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.067447    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 11:05:05.082378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.082327    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podStartSLOduration=2.082310118 podStartE2EDuration="2.082310118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:05.08197112 +0000 UTC m=+2135.287553187" watchObservedRunningTime="2026-11-03 11:05:05.082310118 +0000 UTC m=+2135.287892171"
Nov 03 11:05:05.116652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.116045    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" podStartSLOduration=2.116029072 podStartE2EDuration="2.116029072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:05.112399199 +0000 UTC m=+2135.317981250" watchObservedRunningTime="2026-11-03 11:05:05.116029072 +0000 UTC m=+2135.321611129"
Nov 03 11:05:05.491860 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.491734    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:05.974163 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:05.959139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:06.060749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.056241    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:05:06.060749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.056300    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:06.060749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.056341    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:05:06.060749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.056393    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 11:05:06.063629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.062145    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:05:06.070799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.070347    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 11:05:06.070799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.070401    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 11:05:06.175664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.175632    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:06.184463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.184420    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"]
Nov 03 11:05:06.262673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.261359    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c8cb1f-h7bnr"]
Nov 03 11:05:06.346208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.346178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:06.770607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:06.755277    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:07.031653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.030958    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:05:07.031653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.031012    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 11:05:07.066392 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.066345    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 11:05:07.066984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.066436    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 11:05:07.596595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.587615    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:07.744598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.739945    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:07.865645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.863675    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"]
Nov 03 11:05:07.872602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.870215    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.884201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.883621    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"]
Nov 03 11:05:07.949680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.949652    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b76c62-openshift-default-dockercfg-4s2l6\""
Nov 03 11:05:07.958208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958160    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.958379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958218    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-token\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.958379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958334    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-credential-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.958558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958512    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-envoy\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.958673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958596    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-data\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.958673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958635    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-certs\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.958792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958678    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-podinfo\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.958792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958719    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr7lf\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-kube-api-access-cr7lf\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.958792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.958777    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istiod-ca-cert\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:07.970361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:07.970332    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d36e5e-b748-484f-b41c-c3a5c91e6172" path="/var/lib/kubelet/pods/f7d36e5e-b748-484f-b41c-c3a5c91e6172/volumes"
Nov 03 11:05:08.035597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.031964    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:05:08.043131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.042674    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 11:05:08.060598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.060554    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-podinfo\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.060780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.060625    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr7lf\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-kube-api-access-cr7lf\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.060780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.060679    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istiod-ca-cert\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.060780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.060762    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.060951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.060794    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-token\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.060951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.060913    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-credential-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.061100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.061081    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-envoy\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.061166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.061154    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-data\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.061220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.061191    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-certs\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.061944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.061893    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-certs\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.062205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.062183    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-credential-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.062448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.062432    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.062675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.062661    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istiod-ca-cert\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.063072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.063043    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-data\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.064829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.064809    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-podinfo\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.064952 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.064924    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-envoy\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.066051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.066030    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 11:05:08.066133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.066076    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 11:05:08.069252 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.069226    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-token\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.070006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.069989    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr7lf\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-kube-api-access-cr7lf\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.285600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.272837    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:08.296832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.296508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:08.851213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.841652    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9" podUID="76955372-6a2e-439f-9aa8-34f0603ca75a" containerName="payload-processing" containerID="cri-o://047690af2794bfb195888cdd10148ad31caefe9d1803fc836416fcdfebd2acfb" gracePeriod=25
Nov 03 11:05:08.854810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.854674    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"]
Nov 03 11:05:08.861604 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:05:08.859995    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b38bf7_aad0_494a_9e2a_13dd62b782c7.slice/crio-6714105c3bfe0f360aa36e9897ab2eb32f86e2f32aa02af56e15b469d23aaff3 WatchSource:0}: Error finding container 6714105c3bfe0f360aa36e9897ab2eb32f86e2f32aa02af56e15b469d23aaff3: Status 404 returned error can't find the container with id 6714105c3bfe0f360aa36e9897ab2eb32f86e2f32aa02af56e15b469d23aaff3
Nov 03 11:05:08.867414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.867378    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:08.867541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.867464    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:08.867541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:08.867524    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:09.031436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.030988    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:05:09.031436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.031044    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 11:05:09.049364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.049289    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:09.065655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.065628    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 11:05:09.065783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.065677    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 11:05:09.384865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.383026    5817 generic.go:356] "Generic (PLEG): container finished" podID="76955372-6a2e-439f-9aa8-34f0603ca75a" containerID="047690af2794bfb195888cdd10148ad31caefe9d1803fc836416fcdfebd2acfb" exitCode=0
Nov 03 11:05:09.384865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.383282    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9" event={"ID":"76955372-6a2e-439f-9aa8-34f0603ca75a","Type":"ContainerDied","Data":"047690af2794bfb195888cdd10148ad31caefe9d1803fc836416fcdfebd2acfb"}
Nov 03 11:05:09.406634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.394436    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" event={"ID":"a9b38bf7-aad0-494a-9e2a-13dd62b782c7","Type":"ContainerStarted","Data":"2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05"}
Nov 03 11:05:09.411594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.407184    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" event={"ID":"a9b38bf7-aad0-494a-9e2a-13dd62b782c7","Type":"ContainerStarted","Data":"6714105c3bfe0f360aa36e9897ab2eb32f86e2f32aa02af56e15b469d23aaff3"}
Nov 03 11:05:09.483600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.480082    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podStartSLOduration=2.480062176 podStartE2EDuration="2.480062176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:09.478279771 +0000 UTC m=+2139.683861833" watchObservedRunningTime="2026-11-03 11:05:09.480062176 +0000 UTC m=+2139.685644228"
Nov 03 11:05:09.570011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.569954    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:05:09.670846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.670814    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-kube-api-access-dq7xq\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-kube-api-access-dq7xq\") pod \"76955372-6a2e-439f-9aa8-34f0603ca75a\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") "
Nov 03 11:05:09.671304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.671281    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-provider-credentials\") pod \"76955372-6a2e-439f-9aa8-34f0603ca75a\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") "
Nov 03 11:05:09.671560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.671542    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-plugins-config-volume\") pod \"76955372-6a2e-439f-9aa8-34f0603ca75a\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") "
Nov 03 11:05:09.671781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.671763    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-routing-overlay-volume\") pod \"76955372-6a2e-439f-9aa8-34f0603ca75a\" (UID: \"76955372-6a2e-439f-9aa8-34f0603ca75a\") "
Nov 03 11:05:09.672325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.672286    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-routing-overlay-volume" pod "76955372-6a2e-439f-9aa8-34f0603ca75a" (UID: "76955372-6a2e-439f-9aa8-34f0603ca75a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:09.672780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.672757    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-plugins-config-volume" pod "76955372-6a2e-439f-9aa8-34f0603ca75a" (UID: "76955372-6a2e-439f-9aa8-34f0603ca75a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:09.672881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.672766    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:09.676978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.674622    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-kube-api-access-dq7xq" pod "76955372-6a2e-439f-9aa8-34f0603ca75a" (UID: "76955372-6a2e-439f-9aa8-34f0603ca75a"). InnerVolumeSpecName "kube-api-access-dq7xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:09.676978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.675020    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-provider-credentials" pod "76955372-6a2e-439f-9aa8-34f0603ca75a" (UID: "76955372-6a2e-439f-9aa8-34f0603ca75a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:09.797598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.788563    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76955372-6a2e-439f-9aa8-34f0603ca75a-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:09.797598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.788623    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq7xq\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-kube-api-access-dq7xq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:09.797598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.788641    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76955372-6a2e-439f-9aa8-34f0603ca75a-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:09.963600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:09.960145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:10.041611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.041417    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:05:10.041611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.041475    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 11:05:10.068682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.068647    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 11:05:10.068949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.068920    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 11:05:10.277688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.275697    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:10.277688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.275826    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:05:10.277688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.275866    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 11:05:10.432794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.432748    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"
Nov 03 11:05:10.434680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.434649    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9" event={"ID":"76955372-6a2e-439f-9aa8-34f0603ca75a","Type":"ContainerDied","Data":"bd9c8d84c3a01451660927723f26db727c9d8be851a4c8a49608cfc7822817d4"}
Nov 03 11:05:10.434875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.434860    5817 scope.go:122] "RemoveContainer" containerID="047690af2794bfb195888cdd10148ad31caefe9d1803fc836416fcdfebd2acfb"
Nov 03 11:05:10.511654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.507634    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"]
Nov 03 11:05:10.547672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.546955    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49p4ld9"]
Nov 03 11:05:10.584608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:10.580729    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:11.042612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.037049    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:05:11.042612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.037108    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 11:05:11.084608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.077952    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 03 11:05:11.084608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.078009    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 03 11:05:11.273535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.273496    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:05:11.273775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.273558    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 11:05:11.407682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.396403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:11.922905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.922866    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76955372-6a2e-439f-9aa8-34f0603ca75a" path="/var/lib/kubelet/pods/76955372-6a2e-439f-9aa8-34f0603ca75a/volumes"
Nov 03 11:05:11.974094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:11.974060    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:12.051880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.051850    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:12.053414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.053331    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:12.057166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.057148    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:05:12.070992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.070976    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:12.071437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.071419    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:12.072370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.072351    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:05:12.273930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.273897    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:05:12.274146 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.273947    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 11:05:12.763609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:12.763147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:13.032330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:13.032237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:13.174313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:13.174277    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:13.273559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:13.273527    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:05:13.273766 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:13.273618    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 11:05:13.341747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:13.341650    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"]
Nov 03 11:05:13.342224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:13.342196    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy" containerID="cri-o://4184ea323bcb87cbc887da1289b776174d78df2f869c2223127bbacb23201f81" gracePeriod=30
Nov 03 11:05:13.418036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:13.417997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:13.571501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:13.569984    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:14.151351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:14.151297    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:14.273757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:14.273724    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:05:14.273928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:14.273783    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 11:05:14.752441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:14.747992    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:15.274137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:15.274100    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:05:15.274607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:15.274164    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 11:05:15.317749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:15.317716    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:15.457623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:15.455595    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:15.816238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:15.816205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:16.018622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:16.009919    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:16.274528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:16.274495    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:05:16.274972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:16.274590    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 11:05:16.573152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:16.573061    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:17.294598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:17.288662    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:17.313264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:17.313152    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:17.314093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:17.314071    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:17.318130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:17.316436    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:05:18.090110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:18.090077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:18.636824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:18.636725    5817 generic.go:356] "Generic (PLEG): container finished" podID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerID="4184ea323bcb87cbc887da1289b776174d78df2f869c2223127bbacb23201f81" exitCode=0
Nov 03 11:05:18.636824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:18.636782    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" event={"ID":"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac","Type":"ContainerDied","Data":"4184ea323bcb87cbc887da1289b776174d78df2f869c2223127bbacb23201f81"}
Nov 03 11:05:18.637353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:18.636842    5817 scope.go:122] "RemoveContainer" containerID="7e13ffc7e849e60b43304880b12032ab6722d710256ec111a7136caf7d268848"
Nov 03 11:05:19.060626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.058031    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:19.130295 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.130267    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:05:19.264205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264121    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istiod-ca-cert\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264210    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-certs\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264314    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-kube-api-access-q4qfl\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264357    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-envoy\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264476    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istiod-ca-cert" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:19.264511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264495    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-credential-socket\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264478    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-certs" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:19.264666 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264651    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-podinfo\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264706    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-data\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264763    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-token\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-token\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264777    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-credential-socket" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:19.264879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264826    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-socket\") pod \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\" (UID: \"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac\") "
Nov 03 11:05:19.264938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.264895    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-data" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:19.265073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.265052    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-socket" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:19.265820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.265794    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.265934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.265825    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.265934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.265840    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.265934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.265855    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.265934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.265870    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.267043 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.267017    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-podinfo" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:05:19.267245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.267219    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-envoy" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:19.267361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.267340    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-kube-api-access-q4qfl" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "kube-api-access-q4qfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:19.267403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.267376    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-token" pod "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" (UID: "3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:19.367258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.367232    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-kube-api-access-q4qfl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.367258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.367260    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.367440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.367274    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.367440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.367289    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:19.648905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.648823    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj" event={"ID":"3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac","Type":"ContainerDied","Data":"834b9be1a5ce1dc98b24d1dddfaed74be662909925ee912fd17e14ee6cee1575"}
Nov 03 11:05:19.648905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.648848    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"
Nov 03 11:05:19.648905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.648891    5817 scope.go:122] "RemoveContainer" containerID="4184ea323bcb87cbc887da1289b776174d78df2f869c2223127bbacb23201f81"
Nov 03 11:05:19.674943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.674917    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"]
Nov 03 11:05:19.685781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.685754    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c8cb1f-openshift-default-58c8d48c7-hr2qj"]
Nov 03 11:05:19.746736 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.746711    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:19.899417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:19.899303    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:20.054047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:20.039324    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" path="/var/lib/kubelet/pods/3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac/volumes"
Nov 03 11:05:20.756097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:20.755152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:20.953380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:20.953345    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:21.114647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:21.109426    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:21.776637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:21.776599    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:22.024087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:22.024055    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"]
Nov 03 11:05:22.031192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:22.031118    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7e50bf-4srbb"]
Nov 03 11:05:22.420651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:22.418118    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:22.459008 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:05:22.458764    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:05:23.147795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:23.147760    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:23.918058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:23.918021    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:23.936290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:23.936250    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a24b2a0-d4ad-41dc-981a-5c735541cfa3" path="/var/lib/kubelet/pods/5a24b2a0-d4ad-41dc-981a-5c735541cfa3/volumes"
Nov 03 11:05:25.148799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:25.148764    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:25.746088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:25.746055    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:26.380282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:26.380249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:26.973457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:26.972263    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:27.690048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:27.689997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:28.445595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:28.444429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:29.140457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:29.140407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:29.278670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:29.278624    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:30.033901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:30.033868    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:30.265605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:30.263784    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:31.047494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:31.047107    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:31.059603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:31.059546    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"]
Nov 03 11:05:31.064349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:31.064318    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52613fd0-cl84p"]
Nov 03 11:05:31.681650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:31.681051    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:31.905820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:31.905788    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf8b517-84e7-453f-8d0a-c228f68d629d" path="/var/lib/kubelet/pods/8cf8b517-84e7-453f-8d0a-c228f68d629d/volumes"
Nov 03 11:05:32.349607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:32.344548    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:33.043032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:33.042999    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:33.682332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:33.682295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:34.303502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:34.303456    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:35.042192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:35.042162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:35.177702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:35.177664    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:35.389990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:35.389857    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:35.942702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:35.942671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:36.197464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:36.197382    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:36.885017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:36.884920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:37.561387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:37.561342    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:37.692355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:37.692316    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:37.833782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:37.833701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:38.463332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:38.463295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:39.251211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:39.251177    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:39.998594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:39.997793    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:40.570833 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:40.570796    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:41.205750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:41.205714    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:41.647322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:41.647213    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"]
Nov 03 11:05:42.011243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:42.011210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:42.175263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:42.175226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:42.337400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:42.337308    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:42.694124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:42.693043    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:43.210777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:43.210739    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:43.364031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:43.363996    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:44.066109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:44.058007    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:44.772186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:44.772152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:45.401189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:45.401149    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:46.115887 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:46.115633    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:46.648379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:46.648338    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v" podUID="b642cdd7-025a-460e-907e-2927365d6cb7" containerName="payload-processing" containerID="cri-o://fbe1c51232e0b737e2e511002e756f51e5dd183056a6b3d6b32fe3e7670104fe" gracePeriod=25
Nov 03 11:05:46.780270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:46.780225    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:47.001347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.001321    5817 generic.go:356] "Generic (PLEG): container finished" podID="b642cdd7-025a-460e-907e-2927365d6cb7" containerID="fbe1c51232e0b737e2e511002e756f51e5dd183056a6b3d6b32fe3e7670104fe" exitCode=0
Nov 03 11:05:47.001508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.001410    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v" event={"ID":"b642cdd7-025a-460e-907e-2927365d6cb7","Type":"ContainerDied","Data":"fbe1c51232e0b737e2e511002e756f51e5dd183056a6b3d6b32fe3e7670104fe"}
Nov 03 11:05:47.001508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.001500    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v" event={"ID":"b642cdd7-025a-460e-907e-2927365d6cb7","Type":"ContainerDied","Data":"eae5c5be18de37144828c67b4a0f14d7d294e62abf896fb63bbef8b6685011b7"}
Nov 03 11:05:47.001615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.001522    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae5c5be18de37144828c67b4a0f14d7d294e62abf896fb63bbef8b6685011b7"
Nov 03 11:05:47.011397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.011375    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:05:47.148998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.148963    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-plugins-config-volume\") pod \"b642cdd7-025a-460e-907e-2927365d6cb7\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") "
Nov 03 11:05:47.149759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.149232    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-kube-api-access-gnn8t\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-kube-api-access-gnn8t\") pod \"b642cdd7-025a-460e-907e-2927365d6cb7\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") "
Nov 03 11:05:47.149759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.149474    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-provider-credentials\") pod \"b642cdd7-025a-460e-907e-2927365d6cb7\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") "
Nov 03 11:05:47.149759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.149477    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-plugins-config-volume" pod "b642cdd7-025a-460e-907e-2927365d6cb7" (UID: "b642cdd7-025a-460e-907e-2927365d6cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:47.149759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.149629    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-routing-overlay-volume\") pod \"b642cdd7-025a-460e-907e-2927365d6cb7\" (UID: \"b642cdd7-025a-460e-907e-2927365d6cb7\") "
Nov 03 11:05:47.150120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.150090    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-routing-overlay-volume" pod "b642cdd7-025a-460e-907e-2927365d6cb7" (UID: "b642cdd7-025a-460e-907e-2927365d6cb7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:47.150826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.150802    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:47.150918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.150831    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b642cdd7-025a-460e-907e-2927365d6cb7-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:47.152022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.151996    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-provider-credentials" pod "b642cdd7-025a-460e-907e-2927365d6cb7" (UID: "b642cdd7-025a-460e-907e-2927365d6cb7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:47.152141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.152036    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-kube-api-access-gnn8t" pod "b642cdd7-025a-460e-907e-2927365d6cb7" (UID: "b642cdd7-025a-460e-907e-2927365d6cb7"). InnerVolumeSpecName "kube-api-access-gnn8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:47.252282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.252210    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:47.252282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.252235    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnn8t\" (UniqueName: \"kubernetes.io/projected/b642cdd7-025a-460e-907e-2927365d6cb7-kube-api-access-gnn8t\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:47.598655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.587059    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:47.742600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.728512    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:47.983595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:47.977333    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:48.094862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:48.093474    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"
Nov 03 11:05:48.287058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:48.286975    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"]
Nov 03 11:05:48.300383 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:48.300353    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-bd9cwgr2v"]
Nov 03 11:05:48.349934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:48.349897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:49.063550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:49.063516    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:49.749549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:49.749498    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:49.907608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:49.907539    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b642cdd7-025a-460e-907e-2927365d6cb7" path="/var/lib/kubelet/pods/b642cdd7-025a-460e-907e-2927365d6cb7/volumes"
Nov 03 11:05:50.461351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:50.461308    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:51.074255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:51.074210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:51.672942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:51.672907    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:52.343723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:52.343686    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:52.423223 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:05:52.423183    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:05:53.081599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:53.074892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:53.663848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:53.663813    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:54.270148 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.270040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:54.364339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.364312    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"]
Nov 03 11:05:54.365082 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365068    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b642cdd7-025a-460e-907e-2927365d6cb7" containerName="payload-processing"
Nov 03 11:05:54.365135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365084    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b642cdd7-025a-460e-907e-2927365d6cb7" containerName="payload-processing"
Nov 03 11:05:54.365135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365125    5817 cpu_manager.go:414] "containerMap: removing container" podUID="76955372-6a2e-439f-9aa8-34f0603ca75a" containerName="payload-processing"
Nov 03 11:05:54.365135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365131    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76955372-6a2e-439f-9aa8-34f0603ca75a" containerName="payload-processing"
Nov 03 11:05:54.365245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365152    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy"
Nov 03 11:05:54.365245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365158    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy"
Nov 03 11:05:54.365245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365165    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy"
Nov 03 11:05:54.365245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365170    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy"
Nov 03 11:05:54.365373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365279    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="76955372-6a2e-439f-9aa8-34f0603ca75a" containerName="payload-processing"
Nov 03 11:05:54.365373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365292    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy"
Nov 03 11:05:54.365373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365297    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc7ee5d-3359-4fb2-b5d4-31123a6e54ac" containerName="istio-proxy"
Nov 03 11:05:54.365373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.365305    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b642cdd7-025a-460e-907e-2927365d6cb7" containerName="payload-processing"
Nov 03 11:05:54.368943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.368924    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.384896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.373211    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:05:54.384896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.373762    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:05:54.453705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.417176    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:54.459898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.457777    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"]
Nov 03 11:05:54.508953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.508916    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.509152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.508980    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sf42\" (UniqueName: \"kubernetes.io/projected/078bb7a8-927c-419e-8b08-f73b4da83154-kube-api-access-6sf42\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.509152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.509126    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.610692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.610589    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.610692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.610679    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sf42\" (UniqueName: \"kubernetes.io/projected/078bb7a8-927c-419e-8b08-f73b4da83154-kube-api-access-6sf42\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.610944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.610763    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.611450 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.611424    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.613286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.613260    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.619007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.618978    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sf42\" (UniqueName: \"kubernetes.io/projected/078bb7a8-927c-419e-8b08-f73b4da83154-kube-api-access-6sf42\") pod \"maas-api-revoke-sub-c5a2a639-9lk7g\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:54.744766 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:54.744730    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:55.035676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:55.035642    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"]
Nov 03 11:05:55.037713 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:05:55.037684    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078bb7a8_927c_419e_8b08_f73b4da83154.slice/crio-715c645249b30ef376400b75b4db45d28fe673e0fbaf279d482c92daf103d628 WatchSource:0}: Error finding container 715c645249b30ef376400b75b4db45d28fe673e0fbaf279d482c92daf103d628: Status 404 returned error can't find the container with id 715c645249b30ef376400b75b4db45d28fe673e0fbaf279d482c92daf103d628
Nov 03 11:05:55.152637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:55.147133    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:55.193669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:55.193595    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g" event={"ID":"078bb7a8-927c-419e-8b08-f73b4da83154","Type":"ContainerStarted","Data":"715c645249b30ef376400b75b4db45d28fe673e0fbaf279d482c92daf103d628"}
Nov 03 11:05:55.860628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:55.851538    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:56.204687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:56.204655    5817 generic.go:356] "Generic (PLEG): container finished" podID="078bb7a8-927c-419e-8b08-f73b4da83154" containerID="e2cd4bd39b0311ae9b2d1dbfa6c08a0a02c2790006b67247c128c9c506259ea7" exitCode=0
Nov 03 11:05:56.204879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:56.204738    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g" event={"ID":"078bb7a8-927c-419e-8b08-f73b4da83154","Type":"ContainerDied","Data":"e2cd4bd39b0311ae9b2d1dbfa6c08a0a02c2790006b67247c128c9c506259ea7"}
Nov 03 11:05:56.474655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:56.474148    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:57.412173 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.412146    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:57.513352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.491395    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:57.601839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.601804    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-xks-ca\") pod \"078bb7a8-927c-419e-8b08-f73b4da83154\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") "
Nov 03 11:05:57.602039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.601902    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078bb7a8-927c-419e-8b08-f73b4da83154-kube-api-access-6sf42\" (UniqueName: \"kubernetes.io/projected/078bb7a8-927c-419e-8b08-f73b4da83154-kube-api-access-6sf42\") pod \"078bb7a8-927c-419e-8b08-f73b4da83154\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") "
Nov 03 11:05:57.602039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.601966    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-openshift-ca\") pod \"078bb7a8-927c-419e-8b08-f73b4da83154\" (UID: \"078bb7a8-927c-419e-8b08-f73b4da83154\") "
Nov 03 11:05:57.602410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.602378    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-openshift-ca" pod "078bb7a8-927c-419e-8b08-f73b4da83154" (UID: "078bb7a8-927c-419e-8b08-f73b4da83154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:57.602893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.602872    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:57.604228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.604205    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-xks-ca" pod "078bb7a8-927c-419e-8b08-f73b4da83154" (UID: "078bb7a8-927c-419e-8b08-f73b4da83154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:05:57.604564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.604535    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078bb7a8-927c-419e-8b08-f73b4da83154-kube-api-access-6sf42" pod "078bb7a8-927c-419e-8b08-f73b4da83154" (UID: "078bb7a8-927c-419e-8b08-f73b4da83154"). InnerVolumeSpecName "kube-api-access-6sf42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:57.704432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.704400    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078bb7a8-927c-419e-8b08-f73b4da83154-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:57.704432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:57.704438    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sf42\" (UniqueName: \"kubernetes.io/projected/078bb7a8-927c-419e-8b08-f73b4da83154-kube-api-access-6sf42\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:58.106596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:58.099281    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:58.227751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:58.227722    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"
Nov 03 11:05:58.227928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:58.227721    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g" event={"ID":"078bb7a8-927c-419e-8b08-f73b4da83154","Type":"ContainerDied","Data":"715c645249b30ef376400b75b4db45d28fe673e0fbaf279d482c92daf103d628"}
Nov 03 11:05:58.227928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:58.227874    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715c645249b30ef376400b75b4db45d28fe673e0fbaf279d482c92daf103d628"
Nov 03 11:05:58.377185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:58.377152    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:05:59.079607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:59.078761    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:05:59.762545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:05:59.762502    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:00.259682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:00.259642    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:01.085218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:01.085183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:01.215606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:01.212330    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:01.580595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:01.560536    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:01.824620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:01.823791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:02.363596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:02.360265    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:02.548275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:02.548242    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:02.688824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:02.688793    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:03.149835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:03.149612    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:04.647211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:04.647178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:05.284366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.284332    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"]
Nov 03 11:06:05.285184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.285167    5817 cpu_manager.go:414] "containerMap: removing container" podUID="078bb7a8-927c-419e-8b08-f73b4da83154" containerName="revoke-keys"
Nov 03 11:06:05.285184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.285185    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078bb7a8-927c-419e-8b08-f73b4da83154" containerName="revoke-keys"
Nov 03 11:06:05.285314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.285305    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="078bb7a8-927c-419e-8b08-f73b4da83154" containerName="revoke-keys"
Nov 03 11:06:05.290489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.290461    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.293929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.293906    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:06:05.296725 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.296700    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"]
Nov 03 11:06:05.297023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.296975    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:06:05.323584 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.323554    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.323711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.323691    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.323758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.323747    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vg8\" (UniqueName: \"kubernetes.io/projected/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-kube-api-access-r2vg8\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.390218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.390009    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:05.425379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.425347    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.425602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.425440    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.425602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.425505    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vg8\" (UniqueName: \"kubernetes.io/projected/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-kube-api-access-r2vg8\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.426261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.426189    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.428035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.428014    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.442159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.442130    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vg8\" (UniqueName: \"kubernetes.io/projected/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-kube-api-access-r2vg8\") pod \"maas-api-revoke-sub-ad1e36d6-m74nj\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.612685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.612612    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:05.958476 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:05.958443    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056fbbb6_d2a5_4a7b_bb7d_543af21d909c.slice/crio-17d2d458d8a9098b7fd632ad90e33912f9b63a555473b094b4d3f7b482fe3071 WatchSource:0}: Error finding container 17d2d458d8a9098b7fd632ad90e33912f9b63a555473b094b4d3f7b482fe3071: Status 404 returned error can't find the container with id 17d2d458d8a9098b7fd632ad90e33912f9b63a555473b094b4d3f7b482fe3071
Nov 03 11:06:05.961014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:05.960988    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"]
Nov 03 11:06:06.271675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:06.270986    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:06.438675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:06.438643    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj" event={"ID":"056fbbb6-d2a5-4a7b-bb7d-543af21d909c","Type":"ContainerStarted","Data":"01f2c2de50b7070ca10a1717efc93781eaa16cce93383d5a0c000cbd7bfb21e0"}
Nov 03 11:06:06.438844 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:06.438726    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj" event={"ID":"056fbbb6-d2a5-4a7b-bb7d-543af21d909c","Type":"ContainerStarted","Data":"17d2d458d8a9098b7fd632ad90e33912f9b63a555473b094b4d3f7b482fe3071"}
Nov 03 11:06:06.456300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:06.456246    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj" podStartSLOduration=1.456233648 podStartE2EDuration="1.456233648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:06.454347295 +0000 UTC m=+2196.659929338" watchObservedRunningTime="2026-11-03 11:06:06.456233648 +0000 UTC m=+2196.661815698"
Nov 03 11:06:06.974861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:06.974829    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:07.454694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:07.454664    5817 generic.go:356] "Generic (PLEG): container finished" podID="056fbbb6-d2a5-4a7b-bb7d-543af21d909c" containerID="01f2c2de50b7070ca10a1717efc93781eaa16cce93383d5a0c000cbd7bfb21e0" exitCode=0
Nov 03 11:06:07.454884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:07.454735    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj" event={"ID":"056fbbb6-d2a5-4a7b-bb7d-543af21d909c","Type":"ContainerDied","Data":"01f2c2de50b7070ca10a1717efc93781eaa16cce93383d5a0c000cbd7bfb21e0"}
Nov 03 11:06:07.666500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:07.666403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:08.311982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.311941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:08.635481 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.635458    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:08.783326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.783297    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-xks-ca\") pod \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") "
Nov 03 11:06:08.783519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.783419    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-openshift-ca\") pod \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") "
Nov 03 11:06:08.783618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.783563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-kube-api-access-r2vg8\" (UniqueName: \"kubernetes.io/projected/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-kube-api-access-r2vg8\") pod \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\" (UID: \"056fbbb6-d2a5-4a7b-bb7d-543af21d909c\") "
Nov 03 11:06:08.783922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.783872    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-openshift-ca" pod "056fbbb6-d2a5-4a7b-bb7d-543af21d909c" (UID: "056fbbb6-d2a5-4a7b-bb7d-543af21d909c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:08.784206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.784187    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:08.785735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.785712    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-xks-ca" pod "056fbbb6-d2a5-4a7b-bb7d-543af21d909c" (UID: "056fbbb6-d2a5-4a7b-bb7d-543af21d909c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:06:08.785880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.785859    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-kube-api-access-r2vg8" pod "056fbbb6-d2a5-4a7b-bb7d-543af21d909c" (UID: "056fbbb6-d2a5-4a7b-bb7d-543af21d909c"). InnerVolumeSpecName "kube-api-access-r2vg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:08.886294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.886230    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vg8\" (UniqueName: \"kubernetes.io/projected/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-kube-api-access-r2vg8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:08.886294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:08.886258    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056fbbb6-d2a5-4a7b-bb7d-543af21d909c-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:09.031189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:09.031151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:09.473144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:09.473113    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"
Nov 03 11:06:09.473648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:09.473143    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj" event={"ID":"056fbbb6-d2a5-4a7b-bb7d-543af21d909c","Type":"ContainerDied","Data":"17d2d458d8a9098b7fd632ad90e33912f9b63a555473b094b4d3f7b482fe3071"}
Nov 03 11:06:09.473648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:09.473189    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17d2d458d8a9098b7fd632ad90e33912f9b63a555473b094b4d3f7b482fe3071"
Nov 03 11:06:09.650030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:09.649997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:10.367413 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:10.367375    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:11.047774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:11.047737    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:11.641178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:11.641135    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:12.344116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:12.344080    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:13.138378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:13.138323    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:13.867210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:13.867173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:14.014885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:14.014850    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:14.157911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:14.157823    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:15.053386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:15.053339    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:15.199100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:15.199066    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:16.192832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:16.187101    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:16.890609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:16.886792    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:17.680330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:17.680295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:17.831806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:17.831754    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:18.413592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:18.412286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:19.141612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:19.141162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:19.859043 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:19.859004    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:20.001156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.001124    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:20.171596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.171484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:20.305826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.305797    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"]
Nov 03 11:06:20.307337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.307313    5817 cpu_manager.go:414] "containerMap: removing container" podUID="056fbbb6-d2a5-4a7b-bb7d-543af21d909c" containerName="revoke-keys"
Nov 03 11:06:20.307494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.307484    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056fbbb6-d2a5-4a7b-bb7d-543af21d909c" containerName="revoke-keys"
Nov 03 11:06:20.307749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.307735    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="056fbbb6-d2a5-4a7b-bb7d-543af21d909c" containerName="revoke-keys"
Nov 03 11:06:20.311862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.311843    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.317850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.317830    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"agc-cred-4909c6868a9d\""
Nov 03 11:06:20.318209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.318189    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"]
Nov 03 11:06:20.318687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.318617    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"kube-root-ca.crt\""
Nov 03 11:06:20.318687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.318641    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"openshift-service-ca.crt\""
Nov 03 11:06:20.318687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.318617    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d9f2ce\""
Nov 03 11:06:20.318687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.318665    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"routing-overlay\""
Nov 03 11:06:20.318971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.318698    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"payload-processing-external-model-e2e--f843bc49-dockercfg-lbvq4\""
Nov 03 11:06:20.504013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.503963    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.504446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.504421    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xcl\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-kube-api-access-75xcl\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.504698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.504610    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.504976 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.504959    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.608747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.608702    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.608975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.608882    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xcl\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-kube-api-access-75xcl\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.608975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.608939    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.609108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.609089    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.610010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.609978    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.616906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.616884    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.629730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.629705    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xcl\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-kube-api-access-75xcl\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.631050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.631018    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.635836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.635813    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:20.938657 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:20.938627    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb39bbb_8f66_45a7_8f04_248550be18e7.slice/crio-e2b6c5d7e9dc676c003be99cd1c15dd9f6d981398c47ff9d1d1f936b6c7fc54d WatchSource:0}: Error finding container e2b6c5d7e9dc676c003be99cd1c15dd9f6d981398c47ff9d1d1f936b6c7fc54d: Status 404 returned error can't find the container with id e2b6c5d7e9dc676c003be99cd1c15dd9f6d981398c47ff9d1d1f936b6c7fc54d
Nov 03 11:06:20.938874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:20.938850    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"]
Nov 03 11:06:21.683656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:21.682999    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x" event={"ID":"0eb39bbb-8f66-45a7-8f04-248550be18e7","Type":"ContainerStarted","Data":"e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f"}
Nov 03 11:06:21.683656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:21.683110    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x" event={"ID":"0eb39bbb-8f66-45a7-8f04-248550be18e7","Type":"ContainerStarted","Data":"e2b6c5d7e9dc676c003be99cd1c15dd9f6d981398c47ff9d1d1f936b6c7fc54d"}
Nov 03 11:06:21.687594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:21.684366    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:22.323292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:22.323256    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:22.449614 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:06:22.449544    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:06:22.664955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:22.664874    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:22.819039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:22.818994    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:23.569567 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:23.569479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:23.782618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:23.782478    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:24.464550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:24.464493    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:25.179595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.170913    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:25.240438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.240379    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x" podStartSLOduration=5.240363287 podStartE2EDuration="5.240363287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:21.73861275 +0000 UTC m=+2211.944194803" watchObservedRunningTime="2026-11-03 11:06:25.240363287 +0000 UTC m=+2215.445945340"
Nov 03 11:06:25.242120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.242094    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"]
Nov 03 11:06:25.247139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.247119    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.252979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.252956    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:06:25.253273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.253254    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:06:25.259151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.259129    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"]
Nov 03 11:06:25.278110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.277886    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.281675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.277792    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"]
Nov 03 11:06:25.281811 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.281722    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"]
Nov 03 11:06:25.323520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.323491    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.323682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.323652    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.323898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.323872    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw692\" (UniqueName: \"kubernetes.io/projected/90589240-73ab-41cb-a847-bd57a47a0433-kube-api-access-bw692\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.425749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.425714    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw692\" (UniqueName: \"kubernetes.io/projected/90589240-73ab-41cb-a847-bd57a47a0433-kube-api-access-bw692\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.425905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.425771    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdlhx\" (UniqueName: \"kubernetes.io/projected/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-kube-api-access-hdlhx\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.425905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.425814    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.425905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.425869    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.426086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.426036    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.426157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.426126    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.426620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.426593    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.428647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.428615    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.433556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.433499    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw692\" (UniqueName: \"kubernetes.io/projected/90589240-73ab-41cb-a847-bd57a47a0433-kube-api-access-bw692\") pod \"maas-api-revoke-sub-0888a862-rrm7c\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.527547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.527512    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlhx\" (UniqueName: \"kubernetes.io/projected/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-kube-api-access-hdlhx\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.528063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.527584    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.528063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.527645    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.528364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.528342    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.530132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.530109    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.535353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.535322    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlhx\" (UniqueName: \"kubernetes.io/projected/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-kube-api-access-hdlhx\") pod \"maas-api-revoke-sub-6d5c93f0-m5s4p\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.573399 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.573376    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:25.595638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.595617    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:25.866600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.860837    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:25.944236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.944215    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"]
Nov 03 11:06:25.953340 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:25.953315    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"]
Nov 03 11:06:26.597655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:26.592875    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:26.738616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:26.736534    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c" event={"ID":"90589240-73ab-41cb-a847-bd57a47a0433","Type":"ContainerStarted","Data":"e3737944727525ba5c3b86448c53c5cd9695569fd9fc63cbfb2635f54eb326fa"}
Nov 03 11:06:26.738616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:26.736632    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c" event={"ID":"90589240-73ab-41cb-a847-bd57a47a0433","Type":"ContainerStarted","Data":"192b714666ded867e56001149234099e67ecda3e3f87d9d4a736e90deeb99208"}
Nov 03 11:06:26.738616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:26.738505    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p" event={"ID":"29dd88e4-d5e5-4e33-8d40-0edadb5d114e","Type":"ContainerStarted","Data":"b107e6a73adb514c32c693d030514336566534a88459d2c6a8ad7a2f27135f66"}
Nov 03 11:06:26.738616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:26.738550    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p" event={"ID":"29dd88e4-d5e5-4e33-8d40-0edadb5d114e","Type":"ContainerStarted","Data":"8a5e7ec08a1c75474f6b42ffbda782db71e923845cfd2bcb522b7e5a114f2bfe"}
Nov 03 11:06:26.754177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:26.754133    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c" podStartSLOduration=1.754120742 podStartE2EDuration="1.754120742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:26.750777331 +0000 UTC m=+2216.956359382" watchObservedRunningTime="2026-11-03 11:06:26.754120742 +0000 UTC m=+2216.959702784"
Nov 03 11:06:26.770241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:26.770198    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p" podStartSLOduration=1.770185544 podStartE2EDuration="1.770185544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:26.766438018 +0000 UTC m=+2216.972020073" watchObservedRunningTime="2026-11-03 11:06:26.770185544 +0000 UTC m=+2216.975767588"
Nov 03 11:06:27.275659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:27.271812    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:27.750137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:27.750104    5817 generic.go:356] "Generic (PLEG): container finished" podID="90589240-73ab-41cb-a847-bd57a47a0433" containerID="e3737944727525ba5c3b86448c53c5cd9695569fd9fc63cbfb2635f54eb326fa" exitCode=0
Nov 03 11:06:27.750534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:27.750176    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c" event={"ID":"90589240-73ab-41cb-a847-bd57a47a0433","Type":"ContainerDied","Data":"e3737944727525ba5c3b86448c53c5cd9695569fd9fc63cbfb2635f54eb326fa"}
Nov 03 11:06:27.751921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:27.751898    5817 generic.go:356] "Generic (PLEG): container finished" podID="29dd88e4-d5e5-4e33-8d40-0edadb5d114e" containerID="b107e6a73adb514c32c693d030514336566534a88459d2c6a8ad7a2f27135f66" exitCode=0
Nov 03 11:06:27.752049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:27.751978    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p" event={"ID":"29dd88e4-d5e5-4e33-8d40-0edadb5d114e","Type":"ContainerDied","Data":"b107e6a73adb514c32c693d030514336566534a88459d2c6a8ad7a2f27135f66"}
Nov 03 11:06:27.985599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:27.984749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:28.143625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:28.143485    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:28.705320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:28.705286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:29.014255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.014228    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:29.018262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.018242    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:29.116456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.116425    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-openshift-ca\") pod \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") "
Nov 03 11:06:29.116649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.116488    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90589240-73ab-41cb-a847-bd57a47a0433-kube-api-access-bw692\" (UniqueName: \"kubernetes.io/projected/90589240-73ab-41cb-a847-bd57a47a0433-kube-api-access-bw692\") pod \"90589240-73ab-41cb-a847-bd57a47a0433\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") "
Nov 03 11:06:29.116649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.116507    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-openshift-ca\") pod \"90589240-73ab-41cb-a847-bd57a47a0433\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") "
Nov 03 11:06:29.116760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.116659    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-kube-api-access-hdlhx\" (UniqueName: \"kubernetes.io/projected/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-kube-api-access-hdlhx\") pod \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") "
Nov 03 11:06:29.116760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.116714    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-xks-ca\") pod \"90589240-73ab-41cb-a847-bd57a47a0433\" (UID: \"90589240-73ab-41cb-a847-bd57a47a0433\") "
Nov 03 11:06:29.116760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.116738    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-xks-ca\") pod \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\" (UID: \"29dd88e4-d5e5-4e33-8d40-0edadb5d114e\") "
Nov 03 11:06:29.116923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.116895    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-openshift-ca" pod "29dd88e4-d5e5-4e33-8d40-0edadb5d114e" (UID: "29dd88e4-d5e5-4e33-8d40-0edadb5d114e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:29.117269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.117244    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-openshift-ca" pod "90589240-73ab-41cb-a847-bd57a47a0433" (UID: "90589240-73ab-41cb-a847-bd57a47a0433"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:29.117778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.117752    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:29.117947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.117784    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:29.119394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.119364    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90589240-73ab-41cb-a847-bd57a47a0433-kube-api-access-bw692" pod "90589240-73ab-41cb-a847-bd57a47a0433" (UID: "90589240-73ab-41cb-a847-bd57a47a0433"). InnerVolumeSpecName "kube-api-access-bw692". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:29.119613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.119551    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-kube-api-access-hdlhx" pod "29dd88e4-d5e5-4e33-8d40-0edadb5d114e" (UID: "29dd88e4-d5e5-4e33-8d40-0edadb5d114e"). InnerVolumeSpecName "kube-api-access-hdlhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:29.119798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.119779    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-xks-ca" pod "90589240-73ab-41cb-a847-bd57a47a0433" (UID: "90589240-73ab-41cb-a847-bd57a47a0433"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:06:29.120035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.120012    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-xks-ca" pod "29dd88e4-d5e5-4e33-8d40-0edadb5d114e" (UID: "29dd88e4-d5e5-4e33-8d40-0edadb5d114e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:06:29.220095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.220019    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw692\" (UniqueName: \"kubernetes.io/projected/90589240-73ab-41cb-a847-bd57a47a0433-kube-api-access-bw692\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:29.220095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.220055    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdlhx\" (UniqueName: \"kubernetes.io/projected/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-kube-api-access-hdlhx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:29.220095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.220082    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90589240-73ab-41cb-a847-bd57a47a0433-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:29.220299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.220098    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29dd88e4-d5e5-4e33-8d40-0edadb5d114e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:29.578875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.578779    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:29.781097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.781072    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"
Nov 03 11:06:29.781289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.781073    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c" event={"ID":"90589240-73ab-41cb-a847-bd57a47a0433","Type":"ContainerDied","Data":"192b714666ded867e56001149234099e67ecda3e3f87d9d4a736e90deeb99208"}
Nov 03 11:06:29.781289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.781223    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="192b714666ded867e56001149234099e67ecda3e3f87d9d4a736e90deeb99208"
Nov 03 11:06:29.782731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.782714    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"
Nov 03 11:06:29.782862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.782765    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p" event={"ID":"29dd88e4-d5e5-4e33-8d40-0edadb5d114e","Type":"ContainerDied","Data":"8a5e7ec08a1c75474f6b42ffbda782db71e923845cfd2bcb522b7e5a114f2bfe"}
Nov 03 11:06:29.782862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:29.782811    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a5e7ec08a1c75474f6b42ffbda782db71e923845cfd2bcb522b7e5a114f2bfe"
Nov 03 11:06:30.323253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:30.318484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:30.991490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:30.985814    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:31.776598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:31.768231    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:32.511597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:32.505443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:33.184753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.184711    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:33.319470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.319444    5817 scope.go:122] "RemoveContainer" containerID="9a12dda80f0c07dd821844f4358d7992d82d327a5ac4776ced05a49347380e35"
Nov 03 11:06:33.341402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.341376    5817 scope.go:122] "RemoveContainer" containerID="246d24a4c4569035cb332e684c73d28de56a73cf97f1be6d56a59d2ca9a3e263"
Nov 03 11:06:33.430283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.430251    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:33.566719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.550394    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"]
Nov 03 11:06:33.577594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.571181    5817 cpu_manager.go:414] "containerMap: removing container" podUID="90589240-73ab-41cb-a847-bd57a47a0433" containerName="revoke-keys"
Nov 03 11:06:33.577594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.571213    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90589240-73ab-41cb-a847-bd57a47a0433" containerName="revoke-keys"
Nov 03 11:06:33.577594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.571242    5817 cpu_manager.go:414] "containerMap: removing container" podUID="29dd88e4-d5e5-4e33-8d40-0edadb5d114e" containerName="revoke-keys"
Nov 03 11:06:33.577594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.571253    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29dd88e4-d5e5-4e33-8d40-0edadb5d114e" containerName="revoke-keys"
Nov 03 11:06:33.577594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.571496    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="90589240-73ab-41cb-a847-bd57a47a0433" containerName="revoke-keys"
Nov 03 11:06:33.577594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.571510    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="29dd88e4-d5e5-4e33-8d40-0edadb5d114e" containerName="revoke-keys"
Nov 03 11:06:33.599464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.599432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:33.808693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.649167    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.811034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.716272    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:33.811034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.810037    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"]
Nov 03 11:06:33.884974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.884942    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5pc\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-kube-api-access-rw5pc\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.885145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.885103    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.885190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.885143    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.885238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.885220    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.987265 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.987214    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5pc\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-kube-api-access-rw5pc\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.987541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.987381    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.987541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.987446    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.987811 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.987564    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.988306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.988280    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.988586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.988555    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.990756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.990721    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:33.995822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:33.995799    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5pc\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-kube-api-access-rw5pc\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:34.146282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.146196    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:34.264097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.264050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:34.442199 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:34.442153    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd198af40_8b3b_4ee6_a78d_71e071933832.slice/crio-a24b9ac59d2e77ca6623ccba4343a0c336cb0b33cdc9d777f7bc88918a3b26ca WatchSource:0}: Error finding container a24b9ac59d2e77ca6623ccba4343a0c336cb0b33cdc9d777f7bc88918a3b26ca: Status 404 returned error can't find the container with id a24b9ac59d2e77ca6623ccba4343a0c336cb0b33cdc9d777f7bc88918a3b26ca
Nov 03 11:06:34.444253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.444206    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"]
Nov 03 11:06:34.552223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.552189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:34.690129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.690024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:34.833629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.831457    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:34.853928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.853889    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd" event={"ID":"d198af40-8b3b-4ee6-a78d-71e071933832","Type":"ContainerStarted","Data":"0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81"}
Nov 03 11:06:34.854094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.854010    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd" event={"ID":"d198af40-8b3b-4ee6-a78d-71e071933832","Type":"ContainerStarted","Data":"a24b9ac59d2e77ca6623ccba4343a0c336cb0b33cdc9d777f7bc88918a3b26ca"}
Nov 03 11:06:34.854157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.854145    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:34.874701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:34.874643    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd" podStartSLOduration=1.8746243470000001 podStartE2EDuration="1.874624347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:34.869621501 +0000 UTC m=+2225.075203553" watchObservedRunningTime="2026-11-03 11:06:34.874624347 +0000 UTC m=+2225.080206397"
Nov 03 11:06:35.640321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:35.636665    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:36.277272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.277236    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:36.553651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.530258    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:36.596607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.595678    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"]
Nov 03 11:06:36.612597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.608837    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.660622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.659993    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59585\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-kube-api-access-59585\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.660622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.660105    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.660622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.660177    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.660622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.660383    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.683654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.683626    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"]
Nov 03 11:06:36.689238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.689200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:36.762230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.762200    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.762416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.762312    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59585\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-kube-api-access-59585\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.762416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.762369    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.762416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.762411    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.763143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.763098    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.763270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.763153    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.765022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.765002    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.773883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.773863    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59585\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-kube-api-access-59585\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:36.880058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.879962    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:36.994596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:36.985519    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:37.074598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.074037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:37.195607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.194124    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"]
Nov 03 11:06:37.224599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.222853    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.243599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.227807    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"]
Nov 03 11:06:37.243599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.229304    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:06:37.243599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.238611    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:06:37.298636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.292354    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"]
Nov 03 11:06:37.298636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.295958    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:37.340770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.340506    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.340770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.340711    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5sl\" (UniqueName: \"kubernetes.io/projected/d10fe2b3-0973-4845-ad38-ac0d08658652-kube-api-access-wj5sl\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.341017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.340803    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.540597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.530481    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:37.614644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.604125    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.614644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.604275    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5sl\" (UniqueName: \"kubernetes.io/projected/d10fe2b3-0973-4845-ad38-ac0d08658652-kube-api-access-wj5sl\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.614644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.604320    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.621225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.621032    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.632033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.630999    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.653552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.638194    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5sl\" (UniqueName: \"kubernetes.io/projected/d10fe2b3-0973-4845-ad38-ac0d08658652-kube-api-access-wj5sl\") pod \"maas-api-revoke-sub-61d9fb22-srz5g\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.725311 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:37.724715    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0cfd8e_80ce_4728_afe6_7c8a52c02839.slice/crio-486ceddaeb999fc5d5a11a05f175a982951d2b29644f7e0f07bf3b93b68bcccc WatchSource:0}: Error finding container 486ceddaeb999fc5d5a11a05f175a982951d2b29644f7e0f07bf3b93b68bcccc: Status 404 returned error can't find the container with id 486ceddaeb999fc5d5a11a05f175a982951d2b29644f7e0f07bf3b93b68bcccc
Nov 03 11:06:37.725834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.725374    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"]
Nov 03 11:06:37.886798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.886762    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z" event={"ID":"7a0cfd8e-80ce-4728-afe6-7c8a52c02839","Type":"ContainerStarted","Data":"ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b"}
Nov 03 11:06:37.887013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.886997    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:37.887103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.887042    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z" event={"ID":"7a0cfd8e-80ce-4728-afe6-7c8a52c02839","Type":"ContainerStarted","Data":"486ceddaeb999fc5d5a11a05f175a982951d2b29644f7e0f07bf3b93b68bcccc"}
Nov 03 11:06:37.908150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.908093    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z" podStartSLOduration=1.908076763 podStartE2EDuration="1.908076763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:37.903630644 +0000 UTC m=+2228.109212694" watchObservedRunningTime="2026-11-03 11:06:37.908076763 +0000 UTC m=+2228.113658822"
Nov 03 11:06:37.908767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.908744    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:37.957615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:37.953762    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:38.270230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:38.267457    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"]
Nov 03 11:06:38.657356 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:38.657196    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:38.925145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:38.925060    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g" event={"ID":"d10fe2b3-0973-4845-ad38-ac0d08658652","Type":"ContainerStarted","Data":"42bdb10f5319569bb6847d8ee644844fc569b80fc30c817db4c22ec7d02c2e3a"}
Nov 03 11:06:38.925145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:38.925128    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g" event={"ID":"d10fe2b3-0973-4845-ad38-ac0d08658652","Type":"ContainerStarted","Data":"55f4ed32a7661f57b73368eeacfb4bffa9c0dde07d34ae7bf37ee04628475722"}
Nov 03 11:06:38.957539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:38.957484    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g" podStartSLOduration=1.957466639 podStartE2EDuration="1.957466639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:38.957060325 +0000 UTC m=+2229.162642376" watchObservedRunningTime="2026-11-03 11:06:38.957466639 +0000 UTC m=+2229.163048690"
Nov 03 11:06:38.991084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:38.991050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:39.141610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:39.136140    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:39.624447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:39.624368    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:39.872813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:39.872777    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:39.931256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:39.931228    5817 generic.go:356] "Generic (PLEG): container finished" podID="d10fe2b3-0973-4845-ad38-ac0d08658652" containerID="42bdb10f5319569bb6847d8ee644844fc569b80fc30c817db4c22ec7d02c2e3a" exitCode=0
Nov 03 11:06:39.931660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:39.931309    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g" event={"ID":"d10fe2b3-0973-4845-ad38-ac0d08658652","Type":"ContainerDied","Data":"42bdb10f5319569bb6847d8ee644844fc569b80fc30c817db4c22ec7d02c2e3a"}
Nov 03 11:06:40.211959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.211890    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:40.307076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.307041    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"]
Nov 03 11:06:40.311783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.311767    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.320012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.319986    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"]
Nov 03 11:06:40.355352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.355326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:40.446033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.446007    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.446171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.446067    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbqq\" (UniqueName: \"kubernetes.io/projected/06e695ab-5123-4d8d-a35f-3525b0d114d2-kube-api-access-ppbqq\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.446216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.446169    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.549836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.547791    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbqq\" (UniqueName: \"kubernetes.io/projected/06e695ab-5123-4d8d-a35f-3525b0d114d2-kube-api-access-ppbqq\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.549836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.547867    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.549836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.548165    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.549836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.549010    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.602602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.598054    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.602602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.598241    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbqq\" (UniqueName: \"kubernetes.io/projected/06e695ab-5123-4d8d-a35f-3525b0d114d2-kube-api-access-ppbqq\") pod \"maas-api-revoke-sub-e1540dc2-4dkxt\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.636164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.636126    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:40.649856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.649803    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:40.929119 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:40.929088    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e695ab_5123_4d8d_a35f_3525b0d114d2.slice/crio-e2e00f18800c4845c12ec68f179523ee7f96aec0a975c21515bd99d1efae18e3 WatchSource:0}: Error finding container e2e00f18800c4845c12ec68f179523ee7f96aec0a975c21515bd99d1efae18e3: Status 404 returned error can't find the container with id e2e00f18800c4845c12ec68f179523ee7f96aec0a975c21515bd99d1efae18e3
Nov 03 11:06:40.930119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.930095    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"]
Nov 03 11:06:40.945226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:40.945201    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt" event={"ID":"06e695ab-5123-4d8d-a35f-3525b0d114d2","Type":"ContainerStarted","Data":"e2e00f18800c4845c12ec68f179523ee7f96aec0a975c21515bd99d1efae18e3"}
Nov 03 11:06:41.112940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.112917    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:41.260407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.260371    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10fe2b3-0973-4845-ad38-ac0d08658652-kube-api-access-wj5sl\" (UniqueName: \"kubernetes.io/projected/d10fe2b3-0973-4845-ad38-ac0d08658652-kube-api-access-wj5sl\") pod \"d10fe2b3-0973-4845-ad38-ac0d08658652\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") "
Nov 03 11:06:41.260619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.260596    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-openshift-ca\") pod \"d10fe2b3-0973-4845-ad38-ac0d08658652\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") "
Nov 03 11:06:41.260733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.260712    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-xks-ca\") pod \"d10fe2b3-0973-4845-ad38-ac0d08658652\" (UID: \"d10fe2b3-0973-4845-ad38-ac0d08658652\") "
Nov 03 11:06:41.261107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.261071    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-openshift-ca" pod "d10fe2b3-0973-4845-ad38-ac0d08658652" (UID: "d10fe2b3-0973-4845-ad38-ac0d08658652"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:41.261546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.261523    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:41.287330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.287297    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-xks-ca" pod "d10fe2b3-0973-4845-ad38-ac0d08658652" (UID: "d10fe2b3-0973-4845-ad38-ac0d08658652"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:06:41.287469 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.287444    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10fe2b3-0973-4845-ad38-ac0d08658652-kube-api-access-wj5sl" pod "d10fe2b3-0973-4845-ad38-ac0d08658652" (UID: "d10fe2b3-0973-4845-ad38-ac0d08658652"). InnerVolumeSpecName "kube-api-access-wj5sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:41.369596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.366615    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10fe2b3-0973-4845-ad38-ac0d08658652-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:41.369596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.366659    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5sl\" (UniqueName: \"kubernetes.io/projected/d10fe2b3-0973-4845-ad38-ac0d08658652-kube-api-access-wj5sl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:41.383771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.383734    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:41.956547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.956517    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"
Nov 03 11:06:41.957016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.956514    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g" event={"ID":"d10fe2b3-0973-4845-ad38-ac0d08658652","Type":"ContainerDied","Data":"55f4ed32a7661f57b73368eeacfb4bffa9c0dde07d34ae7bf37ee04628475722"}
Nov 03 11:06:41.957016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.956656    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55f4ed32a7661f57b73368eeacfb4bffa9c0dde07d34ae7bf37ee04628475722"
Nov 03 11:06:41.968606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.960345    5817 generic.go:356] "Generic (PLEG): container finished" podID="06e695ab-5123-4d8d-a35f-3525b0d114d2" containerID="57134b167c76b24699560d01d1d5134be7dbb4770314249f3610b5ed93bb69cb" exitCode=0
Nov 03 11:06:41.968606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.960668    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt" event={"ID":"06e695ab-5123-4d8d-a35f-3525b0d114d2","Type":"ContainerDied","Data":"57134b167c76b24699560d01d1d5134be7dbb4770314249f3610b5ed93bb69cb"}
Nov 03 11:06:41.991873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:41.983445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:42.293378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.293322    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x" podUID="0eb39bbb-8f66-45a7-8f04-248550be18e7" containerName="payload-processing" containerID="cri-o://e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f" gracePeriod=25
Nov 03 11:06:42.596017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.595991    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:42.651523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.651125    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:42.690667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.690633    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-kube-api-access-75xcl\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-kube-api-access-75xcl\") pod \"0eb39bbb-8f66-45a7-8f04-248550be18e7\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") "
Nov 03 11:06:42.690868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.690726    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-provider-credentials\") pod \"0eb39bbb-8f66-45a7-8f04-248550be18e7\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") "
Nov 03 11:06:42.690868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.690844    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-plugins-config-volume\") pod \"0eb39bbb-8f66-45a7-8f04-248550be18e7\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") "
Nov 03 11:06:42.690997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.690966    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-routing-overlay-volume\") pod \"0eb39bbb-8f66-45a7-8f04-248550be18e7\" (UID: \"0eb39bbb-8f66-45a7-8f04-248550be18e7\") "
Nov 03 11:06:42.692427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.692359    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-routing-overlay-volume" pod "0eb39bbb-8f66-45a7-8f04-248550be18e7" (UID: "0eb39bbb-8f66-45a7-8f04-248550be18e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:42.693218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.693192    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-plugins-config-volume" pod "0eb39bbb-8f66-45a7-8f04-248550be18e7" (UID: "0eb39bbb-8f66-45a7-8f04-248550be18e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:42.696239 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.696198    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-kube-api-access-75xcl" pod "0eb39bbb-8f66-45a7-8f04-248550be18e7" (UID: "0eb39bbb-8f66-45a7-8f04-248550be18e7"). InnerVolumeSpecName "kube-api-access-75xcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:42.703815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.703787    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-provider-credentials" pod "0eb39bbb-8f66-45a7-8f04-248550be18e7" (UID: "0eb39bbb-8f66-45a7-8f04-248550be18e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:42.793909 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.793880    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:42.794059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.793918    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eb39bbb-8f66-45a7-8f04-248550be18e7-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:42.794059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.793928    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75xcl\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-kube-api-access-75xcl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:42.794059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.793937    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eb39bbb-8f66-45a7-8f04-248550be18e7-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:42.973723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.973692    5817 generic.go:356] "Generic (PLEG): container finished" podID="0eb39bbb-8f66-45a7-8f04-248550be18e7" containerID="e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f" exitCode=0
Nov 03 11:06:42.974294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.973750    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"
Nov 03 11:06:42.974294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.974174    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x" event={"ID":"0eb39bbb-8f66-45a7-8f04-248550be18e7","Type":"ContainerDied","Data":"e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f"}
Nov 03 11:06:42.974294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.974228    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x" event={"ID":"0eb39bbb-8f66-45a7-8f04-248550be18e7","Type":"ContainerDied","Data":"e2b6c5d7e9dc676c003be99cd1c15dd9f6d981398c47ff9d1d1f936b6c7fc54d"}
Nov 03 11:06:42.974294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.974258    5817 scope.go:122] "RemoveContainer" containerID="e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f"
Nov 03 11:06:42.989536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.989509    5817 scope.go:122] "RemoveContainer" containerID="e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f"
Nov 03 11:06:42.989888 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:06:42.989862    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f\": container with ID starting with e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f not found: ID does not exist" containerID="e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f"
Nov 03 11:06:42.989998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:42.989899    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f"} err="failed to get container status \"e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f\": rpc error: code = NotFound desc = could not find container \"e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f\": container with ID starting with e19e32e41c3fd7b05c75c1fd0bda77e48286934f1d5950884b50abf44bddc45f not found: ID does not exist"
Nov 03 11:06:43.003092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.003067    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"]
Nov 03 11:06:43.008109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.008085    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd758b4x"]
Nov 03 11:06:43.164493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.164471    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:43.251333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.251238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:43.330600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.304178    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-xks-ca\") pod \"06e695ab-5123-4d8d-a35f-3525b0d114d2\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") "
Nov 03 11:06:43.332643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.331700    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e695ab-5123-4d8d-a35f-3525b0d114d2-kube-api-access-ppbqq\" (UniqueName: \"kubernetes.io/projected/06e695ab-5123-4d8d-a35f-3525b0d114d2-kube-api-access-ppbqq\") pod \"06e695ab-5123-4d8d-a35f-3525b0d114d2\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") "
Nov 03 11:06:43.332643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.331958    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-openshift-ca\") pod \"06e695ab-5123-4d8d-a35f-3525b0d114d2\" (UID: \"06e695ab-5123-4d8d-a35f-3525b0d114d2\") "
Nov 03 11:06:43.333484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.333459    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-openshift-ca" pod "06e695ab-5123-4d8d-a35f-3525b0d114d2" (UID: "06e695ab-5123-4d8d-a35f-3525b0d114d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:43.392849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.392787    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e695ab-5123-4d8d-a35f-3525b0d114d2-kube-api-access-ppbqq" pod "06e695ab-5123-4d8d-a35f-3525b0d114d2" (UID: "06e695ab-5123-4d8d-a35f-3525b0d114d2"). InnerVolumeSpecName "kube-api-access-ppbqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:43.398774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.398715    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-xks-ca" pod "06e695ab-5123-4d8d-a35f-3525b0d114d2" (UID: "06e695ab-5123-4d8d-a35f-3525b0d114d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:06:43.442599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.435082    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:43.442599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.435117    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbqq\" (UniqueName: \"kubernetes.io/projected/06e695ab-5123-4d8d-a35f-3525b0d114d2-kube-api-access-ppbqq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:43.442599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.435132    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e695ab-5123-4d8d-a35f-3525b0d114d2-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:43.458120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.458001    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:43.614897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.614867    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:43.752300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.752267    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:43.905392 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.905301    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb39bbb-8f66-45a7-8f04-248550be18e7" path="/var/lib/kubelet/pods/0eb39bbb-8f66-45a7-8f04-248550be18e7/volumes"
Nov 03 11:06:43.985730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.985700    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"
Nov 03 11:06:43.985730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.985727    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt" event={"ID":"06e695ab-5123-4d8d-a35f-3525b0d114d2","Type":"ContainerDied","Data":"e2e00f18800c4845c12ec68f179523ee7f96aec0a975c21515bd99d1efae18e3"}
Nov 03 11:06:43.986159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:43.985768    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e00f18800c4845c12ec68f179523ee7f96aec0a975c21515bd99d1efae18e3"
Nov 03 11:06:44.082742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:44.082167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:44.799598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:44.797649    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:45.480705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:45.480667    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:45.632249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:45.632189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:45.845643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:45.845455    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:45.869284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:45.869259    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:06:45.922262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:45.922219    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"]
Nov 03 11:06:46.278870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:46.277243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:46.933144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:46.930302    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:47.668280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:47.668243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:48.270832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.270801    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:48.357869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.354376    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"]
Nov 03 11:06:48.399690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.399074    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:48.417884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.417797    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:48.829114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.829085    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:48.906539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.906266    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"]
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930126    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d10fe2b3-0973-4845-ad38-ac0d08658652" containerName="revoke-keys"
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930160    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10fe2b3-0973-4845-ad38-ac0d08658652" containerName="revoke-keys"
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930239    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0eb39bbb-8f66-45a7-8f04-248550be18e7" containerName="payload-processing"
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930249    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb39bbb-8f66-45a7-8f04-248550be18e7" containerName="payload-processing"
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930307    5817 cpu_manager.go:414] "containerMap: removing container" podUID="06e695ab-5123-4d8d-a35f-3525b0d114d2" containerName="revoke-keys"
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930315    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e695ab-5123-4d8d-a35f-3525b0d114d2" containerName="revoke-keys"
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930847    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb39bbb-8f66-45a7-8f04-248550be18e7" containerName="payload-processing"
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930874    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e695ab-5123-4d8d-a35f-3525b0d114d2" containerName="revoke-keys"
Nov 03 11:06:48.938856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:48.930904    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10fe2b3-0973-4845-ad38-ac0d08658652" containerName="revoke-keys"
Nov 03 11:06:49.042626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.042595    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"]
Nov 03 11:06:49.042962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.042946    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.050715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.050520    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:06:49.050926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.050908    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:06:49.089533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.089446    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7jg\" (UniqueName: \"kubernetes.io/projected/8713a290-7ad4-464e-9ff5-be42069ce4b6-kube-api-access-4q7jg\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.089754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.089640    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.090001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.089948    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.191437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.191388    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7jg\" (UniqueName: \"kubernetes.io/projected/8713a290-7ad4-464e-9ff5-be42069ce4b6-kube-api-access-4q7jg\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.191610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.191515    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.191768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.191751    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.192423 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.192395    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.194460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.194434    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.200304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.200277    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7jg\" (UniqueName: \"kubernetes.io/projected/8713a290-7ad4-464e-9ff5-be42069ce4b6-kube-api-access-4q7jg\") pod \"maas-api-revoke-sub-f6cea063-8nrdh\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.376852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.376787    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:49.580692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.580432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:49.671361 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:49.671325    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8713a290_7ad4_464e_9ff5_be42069ce4b6.slice/crio-b46e4270fcd4fbb531ccaf2cf74265c959737b999bc34f69c24ac94642bf02e9 WatchSource:0}: Error finding container b46e4270fcd4fbb531ccaf2cf74265c959737b999bc34f69c24ac94642bf02e9: Status 404 returned error can't find the container with id b46e4270fcd4fbb531ccaf2cf74265c959737b999bc34f69c24ac94642bf02e9
Nov 03 11:06:49.672866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.672837    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"]
Nov 03 11:06:49.972275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:49.972245    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:50.069379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:50.069342    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh" event={"ID":"8713a290-7ad4-464e-9ff5-be42069ce4b6","Type":"ContainerStarted","Data":"b46e4270fcd4fbb531ccaf2cf74265c959737b999bc34f69c24ac94642bf02e9"}
Nov 03 11:06:50.125213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:50.125184    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:50.281662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:50.281393    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:50.434608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:50.429385    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:50.696171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:50.691677    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:50.835303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:50.835273    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:50.923333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:50.923291    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x" podUID="198ace54-2e55-4014-92c8-3b409bae9f96" containerName="payload-processing" containerID="cri-o://43bc921b58a71b24886981b5f3c44cb61b47dc35c77bb13be8d270b3a83b53cf" gracePeriod=25
Nov 03 11:06:50.979219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:50.979173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:51.082927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.082896    5817 generic.go:356] "Generic (PLEG): container finished" podID="8713a290-7ad4-464e-9ff5-be42069ce4b6" containerID="af6836686633f903f19d6fb1e35960bb5d86b9564ed2bd6f6176457fad228c5b" exitCode=0
Nov 03 11:06:51.083198 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.083049    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh" event={"ID":"8713a290-7ad4-464e-9ff5-be42069ce4b6","Type":"ContainerDied","Data":"af6836686633f903f19d6fb1e35960bb5d86b9564ed2bd6f6176457fad228c5b"}
Nov 03 11:06:51.085129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.085105    5817 generic.go:356] "Generic (PLEG): container finished" podID="198ace54-2e55-4014-92c8-3b409bae9f96" containerID="43bc921b58a71b24886981b5f3c44cb61b47dc35c77bb13be8d270b3a83b53cf" exitCode=0
Nov 03 11:06:51.085230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.085165    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x" event={"ID":"198ace54-2e55-4014-92c8-3b409bae9f96","Type":"ContainerDied","Data":"43bc921b58a71b24886981b5f3c44cb61b47dc35c77bb13be8d270b3a83b53cf"}
Nov 03 11:06:51.250332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.226110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:51.350594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.350022    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:06:51.426595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.419169    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-routing-overlay-volume\") pod \"198ace54-2e55-4014-92c8-3b409bae9f96\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") "
Nov 03 11:06:51.426595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.419312    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-plugins-config-volume\") pod \"198ace54-2e55-4014-92c8-3b409bae9f96\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") "
Nov 03 11:06:51.426595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.419490    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-kube-api-access-gxtvn\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-kube-api-access-gxtvn\") pod \"198ace54-2e55-4014-92c8-3b409bae9f96\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") "
Nov 03 11:06:51.426595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.419593    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-provider-credentials\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-provider-credentials\") pod \"198ace54-2e55-4014-92c8-3b409bae9f96\" (UID: \"198ace54-2e55-4014-92c8-3b409bae9f96\") "
Nov 03 11:06:51.460692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.448850    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-routing-overlay-volume" pod "198ace54-2e55-4014-92c8-3b409bae9f96" (UID: "198ace54-2e55-4014-92c8-3b409bae9f96"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:51.460692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.452785    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-plugins-config-volume" pod "198ace54-2e55-4014-92c8-3b409bae9f96" (UID: "198ace54-2e55-4014-92c8-3b409bae9f96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:51.460692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.460412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:51.464421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.464373    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-kube-api-access-gxtvn" pod "198ace54-2e55-4014-92c8-3b409bae9f96" (UID: "198ace54-2e55-4014-92c8-3b409bae9f96"). InnerVolumeSpecName "kube-api-access-gxtvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:51.469250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.469207    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-provider-credentials" pod "198ace54-2e55-4014-92c8-3b409bae9f96" (UID: "198ace54-2e55-4014-92c8-3b409bae9f96"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:51.552607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.539123    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:51.552607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.539162    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198ace54-2e55-4014-92c8-3b409bae9f96-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:51.552607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.539176    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxtvn\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-kube-api-access-gxtvn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:51.552607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.539192    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198ace54-2e55-4014-92c8-3b409bae9f96-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:51.722326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.715224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:51.879234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:51.867402    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:52.005972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.005868    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:52.028044 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.028015    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"]
Nov 03 11:06:52.028887 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.028869    5817 cpu_manager.go:414] "containerMap: removing container" podUID="198ace54-2e55-4014-92c8-3b409bae9f96" containerName="payload-processing"
Nov 03 11:06:52.028887 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.028884    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198ace54-2e55-4014-92c8-3b409bae9f96" containerName="payload-processing"
Nov 03 11:06:52.029086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.029018    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="198ace54-2e55-4014-92c8-3b409bae9f96" containerName="payload-processing"
Nov 03 11:06:52.033381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.033363    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.041737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.041714    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"]
Nov 03 11:06:52.165597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.162727    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjrz\" (UniqueName: \"kubernetes.io/projected/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-kube-api-access-fpjrz\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.165597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.162969    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.165597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.163167    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.273651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.272626    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjrz\" (UniqueName: \"kubernetes.io/projected/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-kube-api-access-fpjrz\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.273651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.272827    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.273651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.272998    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.340783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.329860    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.438349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.434418    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:52.438349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.437073    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"
Nov 03 11:06:52.438349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.437805    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x" event={"ID":"198ace54-2e55-4014-92c8-3b409bae9f96","Type":"ContainerDied","Data":"14edeaa3cea193b02f67ffb7d1f3ff5ff776672615e9710f54f5ecd21b65503d"}
Nov 03 11:06:52.438349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.437863    5817 scope.go:122] "RemoveContainer" containerID="43bc921b58a71b24886981b5f3c44cb61b47dc35c77bb13be8d270b3a83b53cf"
Nov 03 11:06:52.458973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.458907    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjrz\" (UniqueName: \"kubernetes.io/projected/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-kube-api-access-fpjrz\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.471298 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.471273    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f76f7ab9-ddj9v\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.492803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.492511    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"]
Nov 03 11:06:52.502214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.502189    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-7556wmd6x"]
Nov 03 11:06:52.513051 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:06:52.512996    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:06:52.725171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.723932    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:52.772598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.771788    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:52.954983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.954935    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:52.996599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.992897    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"]
Nov 03 11:06:53.003631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:52.999857    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.003631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.001855    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"]
Nov 03 11:06:53.013231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.005062    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"agc-cred-0679b800e960\""
Nov 03 11:06:53.013231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.008362    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"kube-root-ca.crt\""
Nov 03 11:06:53.013231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.008427    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"routing-overlay\""
Nov 03 11:06:53.013231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.008496    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"openshift-service-ca.crt\""
Nov 03 11:06:53.013231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.008546    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9f53bb\""
Nov 03 11:06:53.080897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.080870    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:53.112353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.108680    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-e2e--b43e39c2-dockercfg-6rngp\""
Nov 03 11:06:53.112353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.108804    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"agc-cred-8d95d0be60f9\""
Nov 03 11:06:53.153650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.153610    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.153840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.153659    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.153909 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.153836    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.154279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.154253    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvt4j\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-kube-api-access-fvt4j\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.255077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255042    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-openshift-ca\") pod \"8713a290-7ad4-464e-9ff5-be42069ce4b6\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") "
Nov 03 11:06:53.255322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255126    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8713a290-7ad4-464e-9ff5-be42069ce4b6-kube-api-access-4q7jg\" (UniqueName: \"kubernetes.io/projected/8713a290-7ad4-464e-9ff5-be42069ce4b6-kube-api-access-4q7jg\") pod \"8713a290-7ad4-464e-9ff5-be42069ce4b6\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") "
Nov 03 11:06:53.255322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255200    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-xks-ca\") pod \"8713a290-7ad4-464e-9ff5-be42069ce4b6\" (UID: \"8713a290-7ad4-464e-9ff5-be42069ce4b6\") "
Nov 03 11:06:53.255502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:53.255639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255565    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-openshift-ca" pod "8713a290-7ad4-464e-9ff5-be42069ce4b6" (UID: "8713a290-7ad4-464e-9ff5-be42069ce4b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:53.255639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255604    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.255775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255659    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.255775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255720    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.256011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.255986    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvt4j\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-kube-api-access-fvt4j\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.256362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.256242    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:53.256462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.256419    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.257072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.257046    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.257861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.257839    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-xks-ca" pod "8713a290-7ad4-464e-9ff5-be42069ce4b6" (UID: "8713a290-7ad4-464e-9ff5-be42069ce4b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:06:53.258213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.258187    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8713a290-7ad4-464e-9ff5-be42069ce4b6-kube-api-access-4q7jg" pod "8713a290-7ad4-464e-9ff5-be42069ce4b6" (UID: "8713a290-7ad4-464e-9ff5-be42069ce4b6"). InnerVolumeSpecName "kube-api-access-4q7jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:53.258421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.258407    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.296472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.296446    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvt4j\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-kube-api-access-fvt4j\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.324840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.324813    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"]
Nov 03 11:06:53.326519 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:53.326489    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf505ecce_7e6c_4ab6_8165_1ee72f8dc40e.slice/crio-d9fa2bbcc04d9edd1633e2796c112ee8eb1aaef69281e03b3614ddcce127debb WatchSource:0}: Error finding container d9fa2bbcc04d9edd1633e2796c112ee8eb1aaef69281e03b3614ddcce127debb: Status 404 returned error can't find the container with id d9fa2bbcc04d9edd1633e2796c112ee8eb1aaef69281e03b3614ddcce127debb
Nov 03 11:06:53.355247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.355219    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z" podUID="7a0cfd8e-80ce-4728-afe6-7c8a52c02839" containerName="payload-processing" containerID="cri-o://ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b" gracePeriod=25
Nov 03 11:06:53.357590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.357548    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q7jg\" (UniqueName: \"kubernetes.io/projected/8713a290-7ad4-464e-9ff5-be42069ce4b6-kube-api-access-4q7jg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:53.357670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.357596    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8713a290-7ad4-464e-9ff5-be42069ce4b6-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:53.379421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.379398    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:53.433303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.433271    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:53.766521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.765697    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:53.798520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.797167    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"
Nov 03 11:06:53.798814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.798600    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh" event={"ID":"8713a290-7ad4-464e-9ff5-be42069ce4b6","Type":"ContainerDied","Data":"b46e4270fcd4fbb531ccaf2cf74265c959737b999bc34f69c24ac94642bf02e9"}
Nov 03 11:06:53.798814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.798656    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b46e4270fcd4fbb531ccaf2cf74265c959737b999bc34f69c24ac94642bf02e9"
Nov 03 11:06:53.830708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.830651    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v" event={"ID":"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e","Type":"ContainerStarted","Data":"d9fa2bbcc04d9edd1633e2796c112ee8eb1aaef69281e03b3614ddcce127debb"}
Nov 03 11:06:53.888093 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:53.888062    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedddee25_fe00_433b_887a_a67597b25099.slice/crio-15cbea123bd716f17dda747e2f74d11b3ec140a51a07b4407a94b925e78dba28 WatchSource:0}: Error finding container 15cbea123bd716f17dda747e2f74d11b3ec140a51a07b4407a94b925e78dba28: Status 404 returned error can't find the container with id 15cbea123bd716f17dda747e2f74d11b3ec140a51a07b4407a94b925e78dba28
Nov 03 11:06:53.888336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.888303    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"]
Nov 03 11:06:53.916728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.916377    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198ace54-2e55-4014-92c8-3b409bae9f96" path="/var/lib/kubelet/pods/198ace54-2e55-4014-92c8-3b409bae9f96/volumes"
Nov 03 11:06:53.918033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:53.917634    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:54.047583 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.047550    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:54.096011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.095977    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-routing-overlay-volume\") pod \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") "
Nov 03 11:06:54.096195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.096074    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-kube-api-access-59585\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-kube-api-access-59585\") pod \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") "
Nov 03 11:06:54.096267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.096228    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-plugins-config-volume\") pod \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") "
Nov 03 11:06:54.096333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.096288    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-provider-credentials\") pod \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\" (UID: \"7a0cfd8e-80ce-4728-afe6-7c8a52c02839\") "
Nov 03 11:06:54.096393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.096348    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-routing-overlay-volume" pod "7a0cfd8e-80ce-4728-afe6-7c8a52c02839" (UID: "7a0cfd8e-80ce-4728-afe6-7c8a52c02839"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:54.096980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.096957    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-plugins-config-volume" pod "7a0cfd8e-80ce-4728-afe6-7c8a52c02839" (UID: "7a0cfd8e-80ce-4728-afe6-7c8a52c02839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:54.097154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.096971    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:54.099010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.098974    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-kube-api-access-59585" pod "7a0cfd8e-80ce-4728-afe6-7c8a52c02839" (UID: "7a0cfd8e-80ce-4728-afe6-7c8a52c02839"). InnerVolumeSpecName "kube-api-access-59585". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:54.099815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.099792    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-provider-credentials" pod "7a0cfd8e-80ce-4728-afe6-7c8a52c02839" (UID: "7a0cfd8e-80ce-4728-afe6-7c8a52c02839"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:54.171715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.171685    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:54.198609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.198565    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:54.198609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.198610    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:54.198827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.198620    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59585\" (UniqueName: \"kubernetes.io/projected/7a0cfd8e-80ce-4728-afe6-7c8a52c02839-kube-api-access-59585\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:54.325660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.320438    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:54.608224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.608139    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"]
Nov 03 11:06:54.609402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.609374    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7a0cfd8e-80ce-4728-afe6-7c8a52c02839" containerName="payload-processing"
Nov 03 11:06:54.609402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.609404    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0cfd8e-80ce-4728-afe6-7c8a52c02839" containerName="payload-processing"
Nov 03 11:06:54.609627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.609431    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8713a290-7ad4-464e-9ff5-be42069ce4b6" containerName="revoke-keys"
Nov 03 11:06:54.609627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.609439    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8713a290-7ad4-464e-9ff5-be42069ce4b6" containerName="revoke-keys"
Nov 03 11:06:54.609744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.609688    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0cfd8e-80ce-4728-afe6-7c8a52c02839" containerName="payload-processing"
Nov 03 11:06:54.609744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.609719    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8713a290-7ad4-464e-9ff5-be42069ce4b6" containerName="revoke-keys"
Nov 03 11:06:54.622880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.622558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:54.626300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.624311    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:54.633429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.633406    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"routing-overlay\""
Nov 03 11:06:54.633537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.633503    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"openshift-service-ca.crt\""
Nov 03 11:06:54.633668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.633557    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-917da2\""
Nov 03 11:06:54.634186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.634148    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"kube-root-ca.crt\""
Nov 03 11:06:54.663092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.663035    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"]
Nov 03 11:06:54.664132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.664048    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"payload-processing-external-model-e2e--90f9edc7-dockercfg-pnl4z\""
Nov 03 11:06:54.664132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.664101    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"agc-cred-fc3b5b6056ca\""
Nov 03 11:06:54.874656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.873807    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:54.874656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.873929    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:54.874656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.874155    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:54.874656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.874188    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nqm\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-kube-api-access-m6nqm\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.005472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.977825    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.005472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.977975    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.005472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.983853    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.005472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.991730    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.005472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:54.991774    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nqm\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-kube-api-access-m6nqm\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.005472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.001273    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.065551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.062151    5817 generic.go:356] "Generic (PLEG): container finished" podID="7a0cfd8e-80ce-4728-afe6-7c8a52c02839" containerID="ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b" exitCode=0
Nov 03 11:06:55.065551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.062330    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z" event={"ID":"7a0cfd8e-80ce-4728-afe6-7c8a52c02839","Type":"ContainerDied","Data":"ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b"}
Nov 03 11:06:55.065551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.062379    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z" event={"ID":"7a0cfd8e-80ce-4728-afe6-7c8a52c02839","Type":"ContainerDied","Data":"486ceddaeb999fc5d5a11a05f175a982951d2b29644f7e0f07bf3b93b68bcccc"}
Nov 03 11:06:55.065551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.062408    5817 scope.go:122] "RemoveContainer" containerID="ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b"
Nov 03 11:06:55.065551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.062619    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"
Nov 03 11:06:55.065551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.063765    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.066682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.066658    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nqm\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-kube-api-access-m6nqm\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.089845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.085451    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:55.111792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.107660    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd" event={"ID":"edddee25-fe00-433b-887a-a67597b25099","Type":"ContainerStarted","Data":"50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3"}
Nov 03 11:06:55.111792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.109267    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:06:55.111792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.109306    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd" event={"ID":"edddee25-fe00-433b-887a-a67597b25099","Type":"ContainerStarted","Data":"15cbea123bd716f17dda747e2f74d11b3ec140a51a07b4407a94b925e78dba28"}
Nov 03 11:06:55.153011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.152932    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd" podStartSLOduration=3.15291479 podStartE2EDuration="3.15291479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:55.144124492 +0000 UTC m=+2245.349706543" watchObservedRunningTime="2026-11-03 11:06:55.15291479 +0000 UTC m=+2245.358496843"
Nov 03 11:06:55.154985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.153139    5817 scope.go:122] "RemoveContainer" containerID="ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b"
Nov 03 11:06:55.155649 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:06:55.155620    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b\": container with ID starting with ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b not found: ID does not exist" containerID="ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b"
Nov 03 11:06:55.155845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.155820    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b"} err="failed to get container status \"ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b\": rpc error: code = NotFound desc = could not find container \"ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b\": container with ID starting with ee153b6cef8d4ad76894f015aba91bb09e2f90e9a1b56cf2dbac1ed6b2926c9b not found: ID does not exist"
Nov 03 11:06:55.156963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.156941    5817 generic.go:356] "Generic (PLEG): container finished" podID="f505ecce-7e6c-4ab6-8165-1ee72f8dc40e" containerID="1d25debf0398de6d1144072e7a45dfe2892783d140a4cb7b38feb4b115545d90" exitCode=0
Nov 03 11:06:55.157344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.157326    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v" event={"ID":"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e","Type":"ContainerDied","Data":"1d25debf0398de6d1144072e7a45dfe2892783d140a4cb7b38feb4b115545d90"}
Nov 03 11:06:55.171675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.171639    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"]
Nov 03 11:06:55.180028 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.179997    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7d49x6s5z"]
Nov 03 11:06:55.297437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.297403    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:55.587886 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:06:55.587841    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ec31c1_bf4c_4dd2_9ea3_561b529d3d0c.slice/crio-e47de9ddeb9a3259b3d9394f85d958cf738d342db0a7ea47427e4ba5f5688251 WatchSource:0}: Error finding container e47de9ddeb9a3259b3d9394f85d958cf738d342db0a7ea47427e4ba5f5688251: Status 404 returned error can't find the container with id e47de9ddeb9a3259b3d9394f85d958cf738d342db0a7ea47427e4ba5f5688251
Nov 03 11:06:55.588285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.588264    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"]
Nov 03 11:06:55.905659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.905556    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0cfd8e-80ce-4728-afe6-7c8a52c02839" path="/var/lib/kubelet/pods/7a0cfd8e-80ce-4728-afe6-7c8a52c02839/volumes"
Nov 03 11:06:55.987522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:55.987489    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:06:56.025547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.025517    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:56.252594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.241417    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn" event={"ID":"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c","Type":"ContainerStarted","Data":"7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d"}
Nov 03 11:06:56.252594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.241488    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn" event={"ID":"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c","Type":"ContainerStarted","Data":"e47de9ddeb9a3259b3d9394f85d958cf738d342db0a7ea47427e4ba5f5688251"}
Nov 03 11:06:56.252594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.241820    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:06:56.530410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.530369    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:56.551414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.551361    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn" podStartSLOduration=2.551345238 podStartE2EDuration="2.551345238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:56.27194922 +0000 UTC m=+2246.477531289" watchObservedRunningTime="2026-11-03 11:06:56.551345238 +0000 UTC m=+2246.756927291"
Nov 03 11:06:56.561874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.561840    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-openshift-ca\") pod \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") "
Nov 03 11:06:56.562086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.561919    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-kube-api-access-fpjrz\" (UniqueName: \"kubernetes.io/projected/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-kube-api-access-fpjrz\") pod \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") "
Nov 03 11:06:56.562086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.561986    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-xks-ca\") pod \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\" (UID: \"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e\") "
Nov 03 11:06:56.562292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.562169    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-openshift-ca" pod "f505ecce-7e6c-4ab6-8165-1ee72f8dc40e" (UID: "f505ecce-7e6c-4ab6-8165-1ee72f8dc40e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:56.562916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.562892    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:56.564721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.564691    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-kube-api-access-fpjrz" pod "f505ecce-7e6c-4ab6-8165-1ee72f8dc40e" (UID: "f505ecce-7e6c-4ab6-8165-1ee72f8dc40e"). InnerVolumeSpecName "kube-api-access-fpjrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:56.565158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.565133    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-xks-ca" pod "f505ecce-7e6c-4ab6-8165-1ee72f8dc40e" (UID: "f505ecce-7e6c-4ab6-8165-1ee72f8dc40e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:06:56.664979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.664952    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpjrz\" (UniqueName: \"kubernetes.io/projected/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-kube-api-access-fpjrz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:56.664979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.664978    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:56.868713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:56.868098    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:57.300523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:57.300481    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"
Nov 03 11:06:57.311718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:57.311648    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v" event={"ID":"f505ecce-7e6c-4ab6-8165-1ee72f8dc40e","Type":"ContainerDied","Data":"d9fa2bbcc04d9edd1633e2796c112ee8eb1aaef69281e03b3614ddcce127debb"}
Nov 03 11:06:57.311718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:57.311709    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9fa2bbcc04d9edd1633e2796c112ee8eb1aaef69281e03b3614ddcce127debb"
Nov 03 11:06:58.214648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:58.208786    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:58.976497 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:58.976462    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:06:59.971700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:59.971672    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"]
Nov 03 11:06:59.987086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:06:59.987026    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:00.178597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:00.174053    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:00.635107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:00.635070    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:01.271220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:01.271179    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:02.044552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:02.044516    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:02.679118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:02.679086    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:02.865015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:02.864972    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:03.564821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:03.564784    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:04.247962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:04.247918    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:04.788308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:04.788275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:04.946715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:04.946678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:04.972459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:04.972426    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw" podUID="16535339-e77f-4fd4-8554-447e2e4abc8e" containerName="payload-processing" containerID="cri-o://3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc" gracePeriod=25
Nov 03 11:07:05.255446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.255421    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:07:05.395470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.395389    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-kube-api-access-jdxr5\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-kube-api-access-jdxr5\") pod \"16535339-e77f-4fd4-8554-447e2e4abc8e\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") "
Nov 03 11:07:05.395689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.395561    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-plugins-config-volume\") pod \"16535339-e77f-4fd4-8554-447e2e4abc8e\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") "
Nov 03 11:07:05.395807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.395790    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-provider-credentials\") pod \"16535339-e77f-4fd4-8554-447e2e4abc8e\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") "
Nov 03 11:07:05.395885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.395841    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-routing-overlay-volume\") pod \"16535339-e77f-4fd4-8554-447e2e4abc8e\" (UID: \"16535339-e77f-4fd4-8554-447e2e4abc8e\") "
Nov 03 11:07:05.396037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.396008    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-plugins-config-volume" pod "16535339-e77f-4fd4-8554-447e2e4abc8e" (UID: "16535339-e77f-4fd4-8554-447e2e4abc8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:05.396549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.396473    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-routing-overlay-volume" pod "16535339-e77f-4fd4-8554-447e2e4abc8e" (UID: "16535339-e77f-4fd4-8554-447e2e4abc8e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:05.396806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.396660    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:05.396806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.396683    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16535339-e77f-4fd4-8554-447e2e4abc8e-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:05.397952 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.397926    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-kube-api-access-jdxr5" pod "16535339-e77f-4fd4-8554-447e2e4abc8e" (UID: "16535339-e77f-4fd4-8554-447e2e4abc8e"). InnerVolumeSpecName "kube-api-access-jdxr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:05.398464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.398443    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-provider-credentials" pod "16535339-e77f-4fd4-8554-447e2e4abc8e" (UID: "16535339-e77f-4fd4-8554-447e2e4abc8e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:05.429221 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.429194    5817 generic.go:356] "Generic (PLEG): container finished" podID="16535339-e77f-4fd4-8554-447e2e4abc8e" containerID="3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc" exitCode=0
Nov 03 11:07:05.429337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.429243    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"
Nov 03 11:07:05.429337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.429277    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw" event={"ID":"16535339-e77f-4fd4-8554-447e2e4abc8e","Type":"ContainerDied","Data":"3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc"}
Nov 03 11:07:05.429337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.429333    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw" event={"ID":"16535339-e77f-4fd4-8554-447e2e4abc8e","Type":"ContainerDied","Data":"4590333e9e19c294d5408d8feebc40d6294aef9fac0f8e7663ff55c857bbcae6"}
Nov 03 11:07:05.429466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.429362    5817 scope.go:122] "RemoveContainer" containerID="3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc"
Nov 03 11:07:05.444981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.444960    5817 scope.go:122] "RemoveContainer" containerID="3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc"
Nov 03 11:07:05.445266 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:07:05.445245    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc\": container with ID starting with 3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc not found: ID does not exist" containerID="3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc"
Nov 03 11:07:05.445331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.445278    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc"} err="failed to get container status \"3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc\": rpc error: code = NotFound desc = could not find container \"3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc\": container with ID starting with 3318a3ee4e33fa26c62f4c4d3617b03bc0d00b89de6202f6a3efa888fb1b3cdc not found: ID does not exist"
Nov 03 11:07:05.459452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.459430    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"]
Nov 03 11:07:05.464139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.464119    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-594698ppw"]
Nov 03 11:07:05.498457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.498430    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:05.498544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.498459    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxr5\" (UniqueName: \"kubernetes.io/projected/16535339-e77f-4fd4-8554-447e2e4abc8e-kube-api-access-jdxr5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:05.648163 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.648069    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:05.905491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.905415    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16535339-e77f-4fd4-8554-447e2e4abc8e" path="/var/lib/kubelet/pods/16535339-e77f-4fd4-8554-447e2e4abc8e/volumes"
Nov 03 11:07:05.953341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:05.953295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:06.107063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:06.107028    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:06.226126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:06.226094    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:07:06.280280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:06.280247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:06.444064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:06.444027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:06.700599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:06.699892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:06.866597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:06.863471    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:07.326761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:07.326710    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:07.467857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:07.467821    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:07.608420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:07.608340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:08.216979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:08.216914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:08.320301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:08.320271    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:07:08.880762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:08.878213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:09.503398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:09.503157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:10.216108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:10.216040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:10.982326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:10.982282    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:12.204470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:12.204425    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:12.674515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:12.674418    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:12.982318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:12.982266    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:13.466597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:13.460782    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:13.605848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:13.605803    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:13.958631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:13.958556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:14.150160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:14.150122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:14.512339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:14.512303    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:14.648886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:14.648845    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:14.825650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:14.824193    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:15.048391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.048353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:15.266457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.266421    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"]
Nov 03 11:07:15.267727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.267703    5817 cpu_manager.go:414] "containerMap: removing container" podUID="16535339-e77f-4fd4-8554-447e2e4abc8e" containerName="payload-processing"
Nov 03 11:07:15.267727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.267726    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16535339-e77f-4fd4-8554-447e2e4abc8e" containerName="payload-processing"
Nov 03 11:07:15.267890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.267753    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f505ecce-7e6c-4ab6-8165-1ee72f8dc40e" containerName="revoke-keys"
Nov 03 11:07:15.267890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.267758    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f505ecce-7e6c-4ab6-8165-1ee72f8dc40e" containerName="revoke-keys"
Nov 03 11:07:15.267958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.267894    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="16535339-e77f-4fd4-8554-447e2e4abc8e" containerName="payload-processing"
Nov 03 11:07:15.267958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.267906    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f505ecce-7e6c-4ab6-8165-1ee72f8dc40e" containerName="revoke-keys"
Nov 03 11:07:15.277084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.277060    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.278131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.278109    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"]
Nov 03 11:07:15.280442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.280414    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:07:15.282667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.281662    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:07:15.437605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.437556    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.437899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.437877    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxstp\" (UniqueName: \"kubernetes.io/projected/dc1d31db-ff60-4822-b624-63cf8df3dce8-kube-api-access-jxstp\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.438041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.438017    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.539298 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.539212    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxstp\" (UniqueName: \"kubernetes.io/projected/dc1d31db-ff60-4822-b624-63cf8df3dce8-kube-api-access-jxstp\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.539807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.539308    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.539807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.539460    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.540078 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.540058    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.541807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.541786    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.547553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.547524    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxstp\" (UniqueName: \"kubernetes.io/projected/dc1d31db-ff60-4822-b624-63cf8df3dce8-kube-api-access-jxstp\") pod \"maas-api-revoke-sub-9b917e05-7x9ts\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.594795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.594769    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:15.884613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:15.884586    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"]
Nov 03 11:07:15.884815 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:07:15.884733    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1d31db_ff60_4822_b624_63cf8df3dce8.slice/crio-c2abc51d6da3343431c816d0dc0c0214af84df92f59ba3ab41c7c6d005e3dc5d WatchSource:0}: Error finding container c2abc51d6da3343431c816d0dc0c0214af84df92f59ba3ab41c7c6d005e3dc5d: Status 404 returned error can't find the container with id c2abc51d6da3343431c816d0dc0c0214af84df92f59ba3ab41c7c6d005e3dc5d
Nov 03 11:07:16.068269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:16.068230    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:16.602223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:16.602131    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts" event={"ID":"dc1d31db-ff60-4822-b624-63cf8df3dce8","Type":"ContainerStarted","Data":"c4c71db5697a65b3fad646f0cb19b2e6e34c73d758dd9095151525e01e8ea214"}
Nov 03 11:07:16.602223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:16.602185    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts" event={"ID":"dc1d31db-ff60-4822-b624-63cf8df3dce8","Type":"ContainerStarted","Data":"c2abc51d6da3343431c816d0dc0c0214af84df92f59ba3ab41c7c6d005e3dc5d"}
Nov 03 11:07:16.618765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:16.618709    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts" podStartSLOduration=1.618695399 podStartE2EDuration="1.618695399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:07:16.617548364 +0000 UTC m=+2266.823130414" watchObservedRunningTime="2026-11-03 11:07:16.618695399 +0000 UTC m=+2266.824277450"
Nov 03 11:07:16.764964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:16.764926    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:17.513670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:17.512348    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:17.614305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:17.614274    5817 generic.go:356] "Generic (PLEG): container finished" podID="dc1d31db-ff60-4822-b624-63cf8df3dce8" containerID="c4c71db5697a65b3fad646f0cb19b2e6e34c73d758dd9095151525e01e8ea214" exitCode=0
Nov 03 11:07:17.614729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:17.614352    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts" event={"ID":"dc1d31db-ff60-4822-b624-63cf8df3dce8","Type":"ContainerDied","Data":"c4c71db5697a65b3fad646f0cb19b2e6e34c73d758dd9095151525e01e8ea214"}
Nov 03 11:07:17.802430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:17.802341    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:18.267717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.267681    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:18.806224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.806201    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:18.836560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.835640    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-openshift-ca\") pod \"dc1d31db-ff60-4822-b624-63cf8df3dce8\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") "
Nov 03 11:07:18.836560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.835839    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1d31db-ff60-4822-b624-63cf8df3dce8-kube-api-access-jxstp\" (UniqueName: \"kubernetes.io/projected/dc1d31db-ff60-4822-b624-63cf8df3dce8-kube-api-access-jxstp\") pod \"dc1d31db-ff60-4822-b624-63cf8df3dce8\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") "
Nov 03 11:07:18.836560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.836026    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-xks-ca\") pod \"dc1d31db-ff60-4822-b624-63cf8df3dce8\" (UID: \"dc1d31db-ff60-4822-b624-63cf8df3dce8\") "
Nov 03 11:07:18.836560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.836279    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-openshift-ca" pod "dc1d31db-ff60-4822-b624-63cf8df3dce8" (UID: "dc1d31db-ff60-4822-b624-63cf8df3dce8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:18.836911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.836885    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:18.879112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.839185    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-xks-ca" pod "dc1d31db-ff60-4822-b624-63cf8df3dce8" (UID: "dc1d31db-ff60-4822-b624-63cf8df3dce8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:07:18.879112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.839451    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1d31db-ff60-4822-b624-63cf8df3dce8-kube-api-access-jxstp" pod "dc1d31db-ff60-4822-b624-63cf8df3dce8" (UID: "dc1d31db-ff60-4822-b624-63cf8df3dce8"). InnerVolumeSpecName "kube-api-access-jxstp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:18.890106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.889431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:18.946912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.946872    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxstp\" (UniqueName: \"kubernetes.io/projected/dc1d31db-ff60-4822-b624-63cf8df3dce8-kube-api-access-jxstp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:18.946912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:18.946904    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d31db-ff60-4822-b624-63cf8df3dce8-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:19.668460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:19.668406    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts" event={"ID":"dc1d31db-ff60-4822-b624-63cf8df3dce8","Type":"ContainerDied","Data":"c2abc51d6da3343431c816d0dc0c0214af84df92f59ba3ab41c7c6d005e3dc5d"}
Nov 03 11:07:19.668460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:19.668510    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2abc51d6da3343431c816d0dc0c0214af84df92f59ba3ab41c7c6d005e3dc5d"
Nov 03 11:07:19.668460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:19.668650    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"
Nov 03 11:07:19.683053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:19.683013    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:20.351795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:20.350768    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:20.968600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:20.961733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:21.103309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.103263    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:21.251694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.251605    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:21.664700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.664607    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:21.800601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.800556    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"]
Nov 03 11:07:21.801375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.801357    5817 cpu_manager.go:414] "containerMap: removing container" podUID="dc1d31db-ff60-4822-b624-63cf8df3dce8" containerName="revoke-keys"
Nov 03 11:07:21.801375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.801375    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1d31db-ff60-4822-b624-63cf8df3dce8" containerName="revoke-keys"
Nov 03 11:07:21.801550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.801539    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1d31db-ff60-4822-b624-63cf8df3dce8" containerName="revoke-keys"
Nov 03 11:07:21.805552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.805531    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.818752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.818402    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"]
Nov 03 11:07:21.896353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.896326    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-56687c-openshift-default-dockercfg-flg5x\""
Nov 03 11:07:21.919558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.919478    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dtr\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-kube-api-access-w7dtr\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.919558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.919516    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-token\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.919795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.919638    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-certs\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.919795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.919664    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-credential-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.919795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.919739    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-envoy\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.919954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.919847    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.919954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.919887    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-podinfo\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.919954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.919916    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-data\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:21.920160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:21.920141    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istiod-ca-cert\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.021855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.021820    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istiod-ca-cert\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.021898    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dtr\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-kube-api-access-w7dtr\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.021939    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-token\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022128    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-certs\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022192    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-credential-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022283    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-envoy\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022383 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022372    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022398    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-podinfo\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022428    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-data\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022588 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022477    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-certs\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022621    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istiod-ca-cert\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.022812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.022786    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-credential-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.023072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.023003    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-data\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.023072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.023027    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.024989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.024964    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-envoy\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.025461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.025442    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-podinfo\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.029855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.029829    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-token\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.029977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.029961    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dtr\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-kube-api-access-w7dtr\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.146140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.146106    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:22.381899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.381864    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:22.425202 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:07:22.425128    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:07:22.447339 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:07:22.447311    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce91a647_dc88_49fe_929b_0cccd1b19fa0.slice/crio-b28ff69f70f83e08a599a4c7f84847e8e2b35ea2f0c85d489ebaa825c824098c WatchSource:0}: Error finding container b28ff69f70f83e08a599a4c7f84847e8e2b35ea2f0c85d489ebaa825c824098c: Status 404 returned error can't find the container with id b28ff69f70f83e08a599a4c7f84847e8e2b35ea2f0c85d489ebaa825c824098c
Nov 03 11:07:22.447693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.447671    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"]
Nov 03 11:07:22.450954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.450921    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:22.451038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.450987    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:22.451038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.451022    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:22.728650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.727758    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" event={"ID":"ce91a647-dc88-49fe-929b-0cccd1b19fa0","Type":"ContainerStarted","Data":"a8d5e75f2f0c84767ff023d57db073a7abd3cf9b1ff89b9989b11dc7a402ccf3"}
Nov 03 11:07:22.728650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.727816    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" event={"ID":"ce91a647-dc88-49fe-929b-0cccd1b19fa0","Type":"ContainerStarted","Data":"b28ff69f70f83e08a599a4c7f84847e8e2b35ea2f0c85d489ebaa825c824098c"}
Nov 03 11:07:22.751109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.751052    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podStartSLOduration=1.751038865 podStartE2EDuration="1.751038865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:07:22.746457754 +0000 UTC m=+2272.952039808" watchObservedRunningTime="2026-11-03 11:07:22.751038865 +0000 UTC m=+2272.956620916"
Nov 03 11:07:22.948031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:22.947904    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:23.146772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:23.146698    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:23.148327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:23.148295    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 03 11:07:23.148499 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:23.148349    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 03 11:07:23.638596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:23.638099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:24.146664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:24.146629    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 03 11:07:24.147076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:24.146684    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 03 11:07:24.270618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:24.264897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:24.871597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:24.868056    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:25.147197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:25.147116    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 03 11:07:25.147197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:25.147171    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 03 11:07:25.518631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:25.515097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:26.146922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:26.146888    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 03 11:07:26.147122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:26.146939    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 03 11:07:26.239324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:26.237348    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"]
Nov 03 11:07:26.253640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:26.253525    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:26.406320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:26.406221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:26.919139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:26.919106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:27.147391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:27.147352    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 03 11:07:27.147590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:27.147414    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 03 11:07:27.257132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:27.257096    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:27.620894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:27.620788    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:27.760369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:27.760339    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:27.919448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:27.919332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:28.079721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:28.079680    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:28.147151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:28.147113    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 03 11:07:28.147316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:28.147179    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 03 11:07:28.229339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:28.229301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:28.390599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:28.375053    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:28.568361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:28.568274    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:29.057630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:29.057566    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:29.146792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:29.146761    5817 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 03 11:07:29.146972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:29.146815    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 03 11:07:29.281762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:29.281729    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:29.632205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:29.631970    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:30.496907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:30.496854    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:30.663372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:30.663337    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:30.664796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:30.664764    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:30.666059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:30.666040    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:07:31.227375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.225685    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:31.246754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.246712    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd" podUID="d198af40-8b3b-4ee6-a78d-71e071933832" containerName="payload-processing" containerID="cri-o://0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81" gracePeriod=25
Nov 03 11:07:31.554313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.554289    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:07:31.652367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.652324    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-kube-api-access-rw5pc\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-kube-api-access-rw5pc\") pod \"d198af40-8b3b-4ee6-a78d-71e071933832\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") "
Nov 03 11:07:31.652561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.652480    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-provider-credentials\") pod \"d198af40-8b3b-4ee6-a78d-71e071933832\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") "
Nov 03 11:07:31.652561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.652521    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-plugins-config-volume\") pod \"d198af40-8b3b-4ee6-a78d-71e071933832\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") "
Nov 03 11:07:31.652848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.652826    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-routing-overlay-volume\") pod \"d198af40-8b3b-4ee6-a78d-71e071933832\" (UID: \"d198af40-8b3b-4ee6-a78d-71e071933832\") "
Nov 03 11:07:31.652990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.652965    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-plugins-config-volume" pod "d198af40-8b3b-4ee6-a78d-71e071933832" (UID: "d198af40-8b3b-4ee6-a78d-71e071933832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:31.653285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.653258    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-routing-overlay-volume" pod "d198af40-8b3b-4ee6-a78d-71e071933832" (UID: "d198af40-8b3b-4ee6-a78d-71e071933832"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:31.653670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.653648    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:31.653738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.653676    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198af40-8b3b-4ee6-a78d-71e071933832-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:31.654960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.654932    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-kube-api-access-rw5pc" pod "d198af40-8b3b-4ee6-a78d-71e071933832" (UID: "d198af40-8b3b-4ee6-a78d-71e071933832"). InnerVolumeSpecName "kube-api-access-rw5pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:31.655458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.655435    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-provider-credentials" pod "d198af40-8b3b-4ee6-a78d-71e071933832" (UID: "d198af40-8b3b-4ee6-a78d-71e071933832"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:31.663812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.663784    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:31.757596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.756879    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw5pc\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-kube-api-access-rw5pc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:31.757596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.756913    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d198af40-8b3b-4ee6-a78d-71e071933832-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:31.976396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.976362    5817 generic.go:356] "Generic (PLEG): container finished" podID="d198af40-8b3b-4ee6-a78d-71e071933832" containerID="0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81" exitCode=0
Nov 03 11:07:31.976653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.976413    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"
Nov 03 11:07:31.976653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.976447    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd" event={"ID":"d198af40-8b3b-4ee6-a78d-71e071933832","Type":"ContainerDied","Data":"0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81"}
Nov 03 11:07:31.976653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.976520    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd" event={"ID":"d198af40-8b3b-4ee6-a78d-71e071933832","Type":"ContainerDied","Data":"a24b9ac59d2e77ca6623ccba4343a0c336cb0b33cdc9d777f7bc88918a3b26ca"}
Nov 03 11:07:31.976653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.976554    5817 scope.go:122] "RemoveContainer" containerID="0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81"
Nov 03 11:07:31.980378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.980353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:31.990079 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.990036    5817 scope.go:122] "RemoveContainer" containerID="0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81"
Nov 03 11:07:31.990931 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:07:31.990889    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81\": container with ID starting with 0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81 not found: ID does not exist" containerID="0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81"
Nov 03 11:07:31.991039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:31.990938    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81"} err="failed to get container status \"0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81\": rpc error: code = NotFound desc = could not find container \"0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81\": container with ID starting with 0d4ff7123ae7bdcfee666fae9383f2efe15cdbe5c10004a32884d05eec7deb81 not found: ID does not exist"
Nov 03 11:07:32.010639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:32.010616    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"]
Nov 03 11:07:32.018687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:32.018664    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcb8tbnd"]
Nov 03 11:07:32.157595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:32.156950    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:33.436752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:33.434221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:33.634891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:33.611554    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:33.709863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:33.709826    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:07:33.776297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:33.776165    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:33.906319 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:33.906288    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d198af40-8b3b-4ee6-a78d-71e071933832" path="/var/lib/kubelet/pods/d198af40-8b3b-4ee6-a78d-71e071933832/volumes"
Nov 03 11:07:34.627611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:34.626607    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:35.354949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:35.351627    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:35.978211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:35.978178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:36.701601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:36.701105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:37.301595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:37.296804    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:37.943628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:37.943595    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:38.635548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:38.635500    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:39.201149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:39.201112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:39.354098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:39.354057    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:39.591270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:39.591185    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:39.801718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:39.801478    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:40.084102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:40.083952    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:41.052420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:41.051852    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:41.652528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:41.652432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:42.368196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:42.368157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:42.977604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:42.974169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:43.309382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:43.309298    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:43.649213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:43.648995    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:44.340972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:44.340939    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:45.106538 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.106446    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:45.767605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.766623    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:45.917545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.917518    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"]
Nov 03 11:07:45.918317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.918301    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d198af40-8b3b-4ee6-a78d-71e071933832" containerName="payload-processing"
Nov 03 11:07:45.918364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.918319    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d198af40-8b3b-4ee6-a78d-71e071933832" containerName="payload-processing"
Nov 03 11:07:45.918494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.918484    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d198af40-8b3b-4ee6-a78d-71e071933832" containerName="payload-processing"
Nov 03 11:07:45.924309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.924289    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:45.930850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.930814    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-35936b-openshift-default-dockercfg-8mcbd\""
Nov 03 11:07:45.935985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:45.935053    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"]
Nov 03 11:07:46.016534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.016499    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-envoy\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.016750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.016557    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-data\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.016750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.016660    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-credential-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.016932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.016906    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-certs\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.017047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.017021    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.017183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.017066    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07128b9c-8528-46df-9b95-7de2110c2b6b-istiod-ca-cert\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.017256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.017205    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-podinfo\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.017364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.017347    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnbpz\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-kube-api-access-pnbpz\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.017429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.017383    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-token\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.119752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.119662    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-envoy\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.119758    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-data\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.119804    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-credential-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.119871    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-certs\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.119967    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.120001    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07128b9c-8528-46df-9b95-7de2110c2b6b-istiod-ca-cert\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.120136    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-podinfo\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.120177    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-data\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.120367    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnbpz\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-kube-api-access-pnbpz\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.120463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.120403    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-token\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.121061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.121009    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-certs\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.121061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.121022    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-credential-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.121214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.121090    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.121630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.121609    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07128b9c-8528-46df-9b95-7de2110c2b6b-istiod-ca-cert\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.122726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.122703    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-envoy\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.123349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.123330    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-podinfo\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.129438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.129413    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-token\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.129564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.129543    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnbpz\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-kube-api-access-pnbpz\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.245855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.245821    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:46.402061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.397749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:46.538672 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:07:46.538004    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07128b9c_8528_46df_9b95_7de2110c2b6b.slice/crio-63328049468ae0d6fc947b50ae7b4102066dafdcb250a0642095039fae460867 WatchSource:0}: Error finding container 63328049468ae0d6fc947b50ae7b4102066dafdcb250a0642095039fae460867: Status 404 returned error can't find the container with id 63328049468ae0d6fc947b50ae7b4102066dafdcb250a0642095039fae460867
Nov 03 11:07:46.540870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.540845    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"]
Nov 03 11:07:46.541873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.541836    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:46.541970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.541922    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:46.542010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:46.541969    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:47.094595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:47.093754    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:47.324598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:47.324523    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" event={"ID":"07128b9c-8528-46df-9b95-7de2110c2b6b","Type":"ContainerStarted","Data":"97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b"}
Nov 03 11:07:47.325090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:47.324639    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" event={"ID":"07128b9c-8528-46df-9b95-7de2110c2b6b","Type":"ContainerStarted","Data":"63328049468ae0d6fc947b50ae7b4102066dafdcb250a0642095039fae460867"}
Nov 03 11:07:47.348446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:47.348340    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" podStartSLOduration=2.348322078 podStartE2EDuration="2.348322078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:07:47.34465853 +0000 UTC m=+2297.550240583" watchObservedRunningTime="2026-11-03 11:07:47.348322078 +0000 UTC m=+2297.553904129"
Nov 03 11:07:47.829925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:47.829873    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:48.246973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:48.246935    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:48.248357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:48.248332    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 11:07:48.248492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:48.248386    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 11:07:48.360284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:48.360248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:48.995594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:48.992677    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:49.246977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:49.246893    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 11:07:49.246977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:49.246965    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 11:07:49.600744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:49.600652    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:50.247122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:50.247085    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 11:07:50.247306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:50.247152    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 11:07:50.326197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:50.326161    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:50.985595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:50.984620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:51.246716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:51.246631    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 11:07:51.246716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:51.246690    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 11:07:51.638657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:51.636220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:52.258602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:52.246344    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:52.258602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:52.246936    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 11:07:52.258602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:52.246988    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 11:07:52.451162 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:07:52.451109    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:07:52.885672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:52.885532    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:53.251446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:53.251417    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:53.252001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:53.251978    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:53.252732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:53.252711    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:07:53.493314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:53.491473    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:54.342481 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:54.342420    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:54.752973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:54.752939    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:55.461798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:55.461745    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:56.057264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:56.057229    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:56.695631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:56.695511    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:57.366593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:57.365362    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:58.028595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:58.028190    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:58.671597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:58.667164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:07:59.349641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:07:59.349606    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:00.072512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:00.072480    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:00.395591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:00.395490    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:08:00.877321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:00.877282    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:02.464632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:02.464597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:02.725987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:02.725894    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:03.206608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:03.205195    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:03.966051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:03.966010    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:04.858225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:04.858188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:05.023559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:05.023525    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"]
Nov 03 11:08:05.028631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:05.028602    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf41438-4llbr"]
Nov 03 11:08:05.494602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:05.483473    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:05.904711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:05.904629    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e752beb-cf06-4255-b7ff-01f0d73c7088" path="/var/lib/kubelet/pods/5e752beb-cf06-4255-b7ff-01f0d73c7088/volumes"
Nov 03 11:08:06.168433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:06.168343    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:06.849602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:06.849275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:07.446099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:07.446064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:08.100088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:08.100054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:08.667701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:08.667668    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:09.276197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:09.276165    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:09.972596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:09.971067    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:10.591896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:10.591865    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:11.367523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:11.367488    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:12.052617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:12.051998    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:12.722047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:12.722014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:13.451203 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:13.451166    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:14.142657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:14.142620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:14.742595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:14.738279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:15.024728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:15.024640    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"]
Nov 03 11:08:15.030443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:15.030409    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dbefb-bgzkz"]
Nov 03 11:08:15.367661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:15.362222    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:15.908063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:15.908029    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7e6a4c-56ba-4f43-bfd5-70dab4dc814f" path="/var/lib/kubelet/pods/de7e6a4c-56ba-4f43-bfd5-70dab4dc814f/volumes"
Nov 03 11:08:16.010613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:16.008136    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:16.703098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:16.703014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:17.348592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:17.348542    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:17.849326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:17.849276    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:17.989595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:17.988817    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:18.581611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:18.579652    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:19.453374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:19.453342    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:19.972086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:19.970808    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:20.603129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:20.603088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:20.747364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:20.747329    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:21.264974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:21.264939    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:22.004596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:21.990868    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:22.414253 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:08:22.414162    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:08:22.638610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:22.631094    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:23.294598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:23.294453    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:23.968047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:23.965249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:24.888600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:24.885532    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:25.450143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:25.450110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:26.190956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:26.190922    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:26.767352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:26.767315    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:27.380375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:27.380334    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:27.975701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:27.968122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:28.752202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:28.752168    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:29.358502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:29.358392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:30.055192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:30.055153    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:30.772342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:30.772295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:31.355775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:31.355735    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:32.057604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:32.055443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:32.676237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:32.676202    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:32.894832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:32.894794    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:33.279681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:33.279645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:33.907694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:33.907657    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:34.616015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:34.615981    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:35.941874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:35.941842    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:36.435667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:36.428972    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:36.677695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:36.677662    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:08:37.068532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:37.068493    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:38.427180 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:38.427146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:39.510639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:39.510538    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:40.194226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.194144    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:40.478343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.478259    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:40.636595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.628549    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:40.736654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.735617    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"]
Nov 03 11:08:40.762125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.762091    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.815129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.815094    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"]
Nov 03 11:08:40.822669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.816774    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:08:40.829633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.816831    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:08:40.885320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.884521    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.887225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.887196    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvdr\" (UniqueName: \"kubernetes.io/projected/613437d2-207b-4fad-9300-f594fb5cba7f-kube-api-access-xgvdr\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.888489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.888464    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.990208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.990125    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.990208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.990183    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvdr\" (UniqueName: \"kubernetes.io/projected/613437d2-207b-4fad-9300-f594fb5cba7f-kube-api-access-xgvdr\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.990442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.990386    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.991296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.991265    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.992972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.992941    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:40.994521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.994500    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"]
Nov 03 11:08:40.998979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:40.998947    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvdr\" (UniqueName: \"kubernetes.io/projected/613437d2-207b-4fad-9300-f594fb5cba7f-kube-api-access-xgvdr\") pod \"maas-api-revoke-sub-5dc983c0-4d298\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:41.005444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.005422    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.006757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.006735    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"]
Nov 03 11:08:41.091863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.091831    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.091995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.091968    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.092160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.092140    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmqdb\" (UniqueName: \"kubernetes.io/projected/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-kube-api-access-zmqdb\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.118107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.118084    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:41.196073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.193864    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmqdb\" (UniqueName: \"kubernetes.io/projected/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-kube-api-access-zmqdb\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.196073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.194074    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.196073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.194171    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.196073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.194958    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.214326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.214263    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.217059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.216983    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmqdb\" (UniqueName: \"kubernetes.io/projected/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-kube-api-access-zmqdb\") pod \"maas-api-revoke-sub-43d3625d-r9nq2\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.343936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.343860    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:41.409547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.409473    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"]
Nov 03 11:08:41.546719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.546039    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:41.643238 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:08:41.643210    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b6fd3b_49e4_48ef_bce0_3c758e15dbe6.slice/crio-36dcf3a2a052cb540913d482eb44095c90695da7adcf5e572f548b9ff3fcc237 WatchSource:0}: Error finding container 36dcf3a2a052cb540913d482eb44095c90695da7adcf5e572f548b9ff3fcc237: Status 404 returned error can't find the container with id 36dcf3a2a052cb540913d482eb44095c90695da7adcf5e572f548b9ff3fcc237
Nov 03 11:08:41.647164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:41.647139    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"]
Nov 03 11:08:42.155096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:42.154991    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298" event={"ID":"613437d2-207b-4fad-9300-f594fb5cba7f","Type":"ContainerStarted","Data":"3d9b6b60a98df7884a88c8736c2551e5fd3585049bd866400c89bdb8c2a05dc3"}
Nov 03 11:08:42.155096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:42.155056    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298" event={"ID":"613437d2-207b-4fad-9300-f594fb5cba7f","Type":"ContainerStarted","Data":"c183d36aa59f7c879a0d0d89db63bf63591f9b489ffcf522bd732d0d7034f768"}
Nov 03 11:08:42.156855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:42.156829    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2" event={"ID":"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6","Type":"ContainerStarted","Data":"193df6b0762a2e01a5b3fdf570573197b7009ea976aa0bec32e7431797ad702c"}
Nov 03 11:08:42.157017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:42.156871    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2" event={"ID":"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6","Type":"ContainerStarted","Data":"36dcf3a2a052cb540913d482eb44095c90695da7adcf5e572f548b9ff3fcc237"}
Nov 03 11:08:42.175615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:42.175555    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298" podStartSLOduration=2.17554265 podStartE2EDuration="2.17554265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:08:42.170903876 +0000 UTC m=+2352.376485943" watchObservedRunningTime="2026-11-03 11:08:42.17554265 +0000 UTC m=+2352.381124701"
Nov 03 11:08:42.190480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:42.190432    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2" podStartSLOduration=2.190415629 podStartE2EDuration="2.190415629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:08:42.184613831 +0000 UTC m=+2352.390195896" watchObservedRunningTime="2026-11-03 11:08:42.190415629 +0000 UTC m=+2352.395997681"
Nov 03 11:08:42.328593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:42.327106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:42.949656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:42.947014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:43.167353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:43.167324    5817 generic.go:356] "Generic (PLEG): container finished" podID="09b6fd3b-49e4-48ef-bce0-3c758e15dbe6" containerID="193df6b0762a2e01a5b3fdf570573197b7009ea976aa0bec32e7431797ad702c" exitCode=0
Nov 03 11:08:43.167547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:43.167411    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2" event={"ID":"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6","Type":"ContainerDied","Data":"193df6b0762a2e01a5b3fdf570573197b7009ea976aa0bec32e7431797ad702c"}
Nov 03 11:08:43.169349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:43.169326    5817 generic.go:356] "Generic (PLEG): container finished" podID="613437d2-207b-4fad-9300-f594fb5cba7f" containerID="3d9b6b60a98df7884a88c8736c2551e5fd3585049bd866400c89bdb8c2a05dc3" exitCode=0
Nov 03 11:08:43.169494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:43.169402    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298" event={"ID":"613437d2-207b-4fad-9300-f594fb5cba7f","Type":"ContainerDied","Data":"3d9b6b60a98df7884a88c8736c2551e5fd3585049bd866400c89bdb8c2a05dc3"}
Nov 03 11:08:43.405890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:43.405792    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:43.672807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:43.672699    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:44.351766 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.351649    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:44.513072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.513042    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:44.517493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.517467    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:44.568649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.568614    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-xks-ca\") pod \"613437d2-207b-4fad-9300-f594fb5cba7f\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") "
Nov 03 11:08:44.568835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.568745    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-openshift-ca\") pod \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") "
Nov 03 11:08:44.568835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.568770    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-openshift-ca\") pod \"613437d2-207b-4fad-9300-f594fb5cba7f\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") "
Nov 03 11:08:44.568976 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.568840    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-kube-api-access-zmqdb\" (UniqueName: \"kubernetes.io/projected/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-kube-api-access-zmqdb\") pod \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") "
Nov 03 11:08:44.568976 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.568879    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613437d2-207b-4fad-9300-f594fb5cba7f-kube-api-access-xgvdr\" (UniqueName: \"kubernetes.io/projected/613437d2-207b-4fad-9300-f594fb5cba7f-kube-api-access-xgvdr\") pod \"613437d2-207b-4fad-9300-f594fb5cba7f\" (UID: \"613437d2-207b-4fad-9300-f594fb5cba7f\") "
Nov 03 11:08:44.569091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.569012    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-xks-ca\") pod \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\" (UID: \"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6\") "
Nov 03 11:08:44.569188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.569145    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-openshift-ca" pod "09b6fd3b-49e4-48ef-bce0-3c758e15dbe6" (UID: "09b6fd3b-49e4-48ef-bce0-3c758e15dbe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:08:44.569859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.569832    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-openshift-ca" pod "613437d2-207b-4fad-9300-f594fb5cba7f" (UID: "613437d2-207b-4fad-9300-f594fb5cba7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:08:44.569994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.569946    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:44.569994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.569960    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:44.572162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.572017    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-xks-ca" pod "09b6fd3b-49e4-48ef-bce0-3c758e15dbe6" (UID: "09b6fd3b-49e4-48ef-bce0-3c758e15dbe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:08:44.572162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.572124    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-xks-ca" pod "613437d2-207b-4fad-9300-f594fb5cba7f" (UID: "613437d2-207b-4fad-9300-f594fb5cba7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:08:44.572162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.572136    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613437d2-207b-4fad-9300-f594fb5cba7f-kube-api-access-xgvdr" pod "613437d2-207b-4fad-9300-f594fb5cba7f" (UID: "613437d2-207b-4fad-9300-f594fb5cba7f"). InnerVolumeSpecName "kube-api-access-xgvdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:08:44.572451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.572427    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-kube-api-access-zmqdb" pod "09b6fd3b-49e4-48ef-bce0-3c758e15dbe6" (UID: "09b6fd3b-49e4-48ef-bce0-3c758e15dbe6"). InnerVolumeSpecName "kube-api-access-zmqdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:08:44.671598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.671538    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613437d2-207b-4fad-9300-f594fb5cba7f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:44.671598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.671566    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmqdb\" (UniqueName: \"kubernetes.io/projected/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-kube-api-access-zmqdb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:44.671598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.671597    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvdr\" (UniqueName: \"kubernetes.io/projected/613437d2-207b-4fad-9300-f594fb5cba7f-kube-api-access-xgvdr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:44.671832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.671610    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:44.972674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:44.972552    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:45.188278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:45.188243    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"
Nov 03 11:08:45.188486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:45.188277    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298" event={"ID":"613437d2-207b-4fad-9300-f594fb5cba7f","Type":"ContainerDied","Data":"c183d36aa59f7c879a0d0d89db63bf63591f9b489ffcf522bd732d0d7034f768"}
Nov 03 11:08:45.188486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:45.188335    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c183d36aa59f7c879a0d0d89db63bf63591f9b489ffcf522bd732d0d7034f768"
Nov 03 11:08:45.190310 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:45.190287    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"
Nov 03 11:08:45.190444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:45.190329    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2" event={"ID":"09b6fd3b-49e4-48ef-bce0-3c758e15dbe6","Type":"ContainerDied","Data":"36dcf3a2a052cb540913d482eb44095c90695da7adcf5e572f548b9ff3fcc237"}
Nov 03 11:08:45.190444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:45.190376    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36dcf3a2a052cb540913d482eb44095c90695da7adcf5e572f548b9ff3fcc237"
Nov 03 11:08:45.577598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:45.571407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:46.259216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:46.259178    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:46.962082 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:46.962045    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:47.656597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:47.653620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:48.280966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:48.280928    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:48.906891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:48.906859    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:49.710185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:49.710148    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:50.409597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:50.408151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:51.056128 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:51.056091    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:51.878837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:51.878800    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:52.037702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:52.035406    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:52.351623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:52.351443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:52.546528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:52.546364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:52.627500 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:08:52.626149    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:08:52.891073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:52.879013    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:53.129397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:53.129364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:53.776781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:53.776745    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"]
Nov 03 11:08:54.025599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:54.021396    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:54.747625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:54.747563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:55.375855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:55.375811    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:56.025320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:56.025277    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:56.650256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:56.650223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:58.360086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:58.360040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:58.499110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:58.499058    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:58.769242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:58.769205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:58.788265 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:58.788217    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn" podUID="91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" containerName="payload-processing" containerID="cri-o://7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d" gracePeriod=25
Nov 03 11:08:59.107911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.107877    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:08:59.283610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.283549    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-kube-api-access-m6nqm\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-kube-api-access-m6nqm\") pod \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") "
Nov 03 11:08:59.283819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.283705    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-routing-overlay-volume\") pod \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") "
Nov 03 11:08:59.283882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.283851    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-provider-credentials\") pod \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") "
Nov 03 11:08:59.283967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.283945    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-plugins-config-volume\") pod \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\" (UID: \"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c\") "
Nov 03 11:08:59.284169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.284120    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-routing-overlay-volume" pod "91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" (UID: "91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:08:59.284352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.284329    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-plugins-config-volume" pod "91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" (UID: "91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:08:59.284783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.284765    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:59.284865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.284787    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:59.286166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.286140    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-kube-api-access-m6nqm" pod "91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" (UID: "91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c"). InnerVolumeSpecName "kube-api-access-m6nqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:08:59.286273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.286177    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-provider-credentials" pod "91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" (UID: "91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:08:59.386985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.386922    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:59.386985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.386950    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6nqm\" (UniqueName: \"kubernetes.io/projected/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c-kube-api-access-m6nqm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:59.495603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.495223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:08:59.589364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.589335    5817 generic.go:356] "Generic (PLEG): container finished" podID="91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" containerID="7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d" exitCode=0
Nov 03 11:08:59.589548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.589404    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"
Nov 03 11:08:59.589548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.589428    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn" event={"ID":"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c","Type":"ContainerDied","Data":"7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d"}
Nov 03 11:08:59.589548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.589496    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn" event={"ID":"91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c","Type":"ContainerDied","Data":"e47de9ddeb9a3259b3d9394f85d958cf738d342db0a7ea47427e4ba5f5688251"}
Nov 03 11:08:59.589548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.589522    5817 scope.go:122] "RemoveContainer" containerID="7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d"
Nov 03 11:08:59.604127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.604096    5817 scope.go:122] "RemoveContainer" containerID="7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d"
Nov 03 11:08:59.604418 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:08:59.604397    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d\": container with ID starting with 7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d not found: ID does not exist" containerID="7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d"
Nov 03 11:08:59.604504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.604424    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d"} err="failed to get container status \"7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d\": rpc error: code = NotFound desc = could not find container \"7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d\": container with ID starting with 7f5c47a93b205e71a20d42018a74370d03202e3c2bbe518c16cb85341a782f2d not found: ID does not exist"
Nov 03 11:08:59.621833 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.621810    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"]
Nov 03 11:08:59.630903 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.630877    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657f5chn"]
Nov 03 11:08:59.904525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:08:59.904493    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" path="/var/lib/kubelet/pods/91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c/volumes"
Nov 03 11:09:00.163312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:00.163187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:00.780596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:00.774314    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:01.416635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:01.408409    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:02.035593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:02.026461    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:02.721798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:02.721766    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:02.855496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:02.855461    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:03.018292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:03.018192    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:03.214739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:03.214705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:03.368121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:03.368027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:03.529230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:03.529194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:03.742676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:03.717562    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:03.987934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:03.987899    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:04.155647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.150600    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:04.307595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.307196    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:04.496522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.496491    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"]
Nov 03 11:09:04.497755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.497732    5817 cpu_manager.go:414] "containerMap: removing container" podUID="91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" containerName="payload-processing"
Nov 03 11:09:04.497818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.497757    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" containerName="payload-processing"
Nov 03 11:09:04.497818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.497804    5817 cpu_manager.go:414] "containerMap: removing container" podUID="613437d2-207b-4fad-9300-f594fb5cba7f" containerName="revoke-keys"
Nov 03 11:09:04.497818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.497811    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613437d2-207b-4fad-9300-f594fb5cba7f" containerName="revoke-keys"
Nov 03 11:09:04.497918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.497848    5817 cpu_manager.go:414] "containerMap: removing container" podUID="09b6fd3b-49e4-48ef-bce0-3c758e15dbe6" containerName="revoke-keys"
Nov 03 11:09:04.497918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.497855    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b6fd3b-49e4-48ef-bce0-3c758e15dbe6" containerName="revoke-keys"
Nov 03 11:09:04.498059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.498046    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="613437d2-207b-4fad-9300-f594fb5cba7f" containerName="revoke-keys"
Nov 03 11:09:04.498097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.498071    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b6fd3b-49e4-48ef-bce0-3c758e15dbe6" containerName="revoke-keys"
Nov 03 11:09:04.498097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.498088    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ec31c1-bf4c-4dd2-9ea3-561b529d3d0c" containerName="payload-processing"
Nov 03 11:09:04.504533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.504508    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.504995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.504970    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"]
Nov 03 11:09:04.510189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.510166    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cbeebd\""
Nov 03 11:09:04.510858 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.510830    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"routing-overlay\""
Nov 03 11:09:04.511023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.510912    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"kube-root-ca.crt\""
Nov 03 11:09:04.511023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.510952    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"openshift-service-ca.crt\""
Nov 03 11:09:04.512649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.512550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:04.574186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.574159    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.574306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.574208    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.574373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.574353    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.574462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.574451    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5wf\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-kube-api-access-zb5wf\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.616933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.616898    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"agc-cred-78cf6722727d\""
Nov 03 11:09:04.617324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.617302    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"payload-processing-external-model-e2e--8c972630-dockercfg-mvh5x\""
Nov 03 11:09:04.716597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.707087    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.716597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.707166    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.716597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.707299    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.716597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.707408    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5wf\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-kube-api-access-zb5wf\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.717922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.717895    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.718717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.718697    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.722140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.722115    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.746721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.737757    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5wf\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-kube-api-access-zb5wf\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:04.851599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:04.846694    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:05.226457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:05.226420    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"]
Nov 03 11:09:05.226973 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:09:05.226945    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee7a78e_1676_4dcd_b4a1_b515fd7f3bd5.slice/crio-c28bd9e76a91fd868ee18d5c10887fcb27dac8b9f13c87c940f84c81882f1cec WatchSource:0}: Error finding container c28bd9e76a91fd868ee18d5c10887fcb27dac8b9f13c87c940f84c81882f1cec: Status 404 returned error can't find the container with id c28bd9e76a91fd868ee18d5c10887fcb27dac8b9f13c87c940f84c81882f1cec
Nov 03 11:09:05.396603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:05.394522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:05.939106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:05.939070    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g" event={"ID":"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5","Type":"ContainerStarted","Data":"70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2"}
Nov 03 11:09:05.939271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:05.939133    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g" event={"ID":"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5","Type":"ContainerStarted","Data":"c28bd9e76a91fd868ee18d5c10887fcb27dac8b9f13c87c940f84c81882f1cec"}
Nov 03 11:09:05.939271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:05.939162    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:05.961816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:05.961771    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g" podStartSLOduration=1.961759188 podStartE2EDuration="1.961759188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:05.956656421 +0000 UTC m=+2376.162238472" watchObservedRunningTime="2026-11-03 11:09:05.961759188 +0000 UTC m=+2376.167341239"
Nov 03 11:09:06.011342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:06.011308    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:06.685599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:06.677365    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:07.252038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:07.250048    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:07.947600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:07.947552    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:08.111540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:08.111508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:08.255792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:08.255707    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:08.859040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:08.859007    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:10.043150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:10.043062    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:10.189915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:10.189877    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:10.330628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:10.330487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:10.463936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:10.463897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:10.758311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:10.758276    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:11.536552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.536521    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"]
Nov 03 11:09:11.543007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.542957    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.547265 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.547236    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"]
Nov 03 11:09:11.547923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.547829    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:09:11.549751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.549110    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:09:11.582926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.582898    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.583090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.583072    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhnqt\" (UniqueName: \"kubernetes.io/projected/b0d06c89-d29c-424f-a971-6fddacf7ea29-kube-api-access-lhnqt\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.583171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.583108    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.684664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.684635    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhnqt\" (UniqueName: \"kubernetes.io/projected/b0d06c89-d29c-424f-a971-6fddacf7ea29-kube-api-access-lhnqt\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.684835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.684670    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.684965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.684946    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.685603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.685507    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.687438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.687411    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.693584 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.693543    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhnqt\" (UniqueName: \"kubernetes.io/projected/b0d06c89-d29c-424f-a971-6fddacf7ea29-kube-api-access-lhnqt\") pod \"maas-api-revoke-sub-a1f96033-mhdq8\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:11.741667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.741616    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:11.866296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:11.866212    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:12.156925 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:09:12.156894    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d06c89_d29c_424f_a971_6fddacf7ea29.slice/crio-8c5b4ac931ff283027da48a48210248d970214e96c6df709862da0347f672eaa WatchSource:0}: Error finding container 8c5b4ac931ff283027da48a48210248d970214e96c6df709862da0347f672eaa: Status 404 returned error can't find the container with id 8c5b4ac931ff283027da48a48210248d970214e96c6df709862da0347f672eaa
Nov 03 11:09:12.158027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:12.158002    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"]
Nov 03 11:09:12.206012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:12.205981    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8" event={"ID":"b0d06c89-d29c-424f-a971-6fddacf7ea29","Type":"ContainerStarted","Data":"8c5b4ac931ff283027da48a48210248d970214e96c6df709862da0347f672eaa"}
Nov 03 11:09:12.460600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:12.454362    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:13.161049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:13.159541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:13.220112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:13.220080    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8" event={"ID":"b0d06c89-d29c-424f-a971-6fddacf7ea29","Type":"ContainerStarted","Data":"ab92637dfaf6061fcf63a8289b658f28f43101001969dd4c90925ea3ac0482a6"}
Nov 03 11:09:13.238171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:13.238129    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8" podStartSLOduration=2.23811579 podStartE2EDuration="2.23811579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:13.235232015 +0000 UTC m=+2383.440814066" watchObservedRunningTime="2026-11-03 11:09:13.23811579 +0000 UTC m=+2383.443697841"
Nov 03 11:09:13.777611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:13.775170    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:14.232394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.232355    5817 generic.go:356] "Generic (PLEG): container finished" podID="b0d06c89-d29c-424f-a971-6fddacf7ea29" containerID="ab92637dfaf6061fcf63a8289b658f28f43101001969dd4c90925ea3ac0482a6" exitCode=0
Nov 03 11:09:14.232394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.232377    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8" event={"ID":"b0d06c89-d29c-424f-a971-6fddacf7ea29","Type":"ContainerDied","Data":"ab92637dfaf6061fcf63a8289b658f28f43101001969dd4c90925ea3ac0482a6"}
Nov 03 11:09:14.525111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.525027    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"]
Nov 03 11:09:14.530337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.530308    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.536882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.536857    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"]
Nov 03 11:09:14.663551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.663519    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhj7\" (UniqueName: \"kubernetes.io/projected/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-kube-api-access-5zhj7\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.663783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.663765    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.663908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.663885    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.730600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.708015    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:14.769611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.765909    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.769611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.766107    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhj7\" (UniqueName: \"kubernetes.io/projected/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-kube-api-access-5zhj7\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.769611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.766338    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.769611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.768084    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.774629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.772344    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.777155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.777092    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhj7\" (UniqueName: \"kubernetes.io/projected/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-kube-api-access-5zhj7\") pod \"maas-api-revoke-sub-6a30f38d-4bzk4\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:14.852264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:14.852237    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:15.137496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.137464    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"]
Nov 03 11:09:15.137899 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:09:15.137727    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d5df9d_eea5_475d_ac9a_2e046d2ad737.slice/crio-99849e8932563707be77a2c6051c1093fafeb2c642a5ab9305e56a08d7e79481 WatchSource:0}: Error finding container 99849e8932563707be77a2c6051c1093fafeb2c642a5ab9305e56a08d7e79481: Status 404 returned error can't find the container with id 99849e8932563707be77a2c6051c1093fafeb2c642a5ab9305e56a08d7e79481
Nov 03 11:09:15.245160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.245125    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4" event={"ID":"d7d5df9d-eea5-475d-ac9a-2e046d2ad737","Type":"ContainerStarted","Data":"99849e8932563707be77a2c6051c1093fafeb2c642a5ab9305e56a08d7e79481"}
Nov 03 11:09:15.409908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.409774    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:15.466059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.466034    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:15.607166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.607141    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d06c89-d29c-424f-a971-6fddacf7ea29-kube-api-access-lhnqt\" (UniqueName: \"kubernetes.io/projected/b0d06c89-d29c-424f-a971-6fddacf7ea29-kube-api-access-lhnqt\") pod \"b0d06c89-d29c-424f-a971-6fddacf7ea29\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") "
Nov 03 11:09:15.607301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.607190    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-xks-ca\") pod \"b0d06c89-d29c-424f-a971-6fddacf7ea29\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") "
Nov 03 11:09:15.607445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.607415    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-openshift-ca\") pod \"b0d06c89-d29c-424f-a971-6fddacf7ea29\" (UID: \"b0d06c89-d29c-424f-a971-6fddacf7ea29\") "
Nov 03 11:09:15.607899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.607878    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-openshift-ca" pod "b0d06c89-d29c-424f-a971-6fddacf7ea29" (UID: "b0d06c89-d29c-424f-a971-6fddacf7ea29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:15.608305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.608274    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:15.609719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.609696    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-xks-ca" pod "b0d06c89-d29c-424f-a971-6fddacf7ea29" (UID: "b0d06c89-d29c-424f-a971-6fddacf7ea29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:09:15.609861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.609839    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d06c89-d29c-424f-a971-6fddacf7ea29-kube-api-access-lhnqt" pod "b0d06c89-d29c-424f-a971-6fddacf7ea29" (UID: "b0d06c89-d29c-424f-a971-6fddacf7ea29"). InnerVolumeSpecName "kube-api-access-lhnqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:15.711060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.711028    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhnqt\" (UniqueName: \"kubernetes.io/projected/b0d06c89-d29c-424f-a971-6fddacf7ea29-kube-api-access-lhnqt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:15.711060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.711061    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d06c89-d29c-424f-a971-6fddacf7ea29-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:15.981648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:15.977618    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:16.278650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.276950    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8" event={"ID":"b0d06c89-d29c-424f-a971-6fddacf7ea29","Type":"ContainerDied","Data":"8c5b4ac931ff283027da48a48210248d970214e96c6df709862da0347f672eaa"}
Nov 03 11:09:16.278650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.277012    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c5b4ac931ff283027da48a48210248d970214e96c6df709862da0347f672eaa"
Nov 03 11:09:16.278650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.276959    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"
Nov 03 11:09:16.283667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.279377    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4" event={"ID":"d7d5df9d-eea5-475d-ac9a-2e046d2ad737","Type":"ContainerStarted","Data":"3669ceda47e5b1ca067b5bb29079c2fa5162359213484d6c6c6f435cb5199c4f"}
Nov 03 11:09:16.306700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.304493    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4" podStartSLOduration=2.304455217 podStartE2EDuration="2.304455217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:16.295502398 +0000 UTC m=+2386.501084450" watchObservedRunningTime="2026-11-03 11:09:16.304455217 +0000 UTC m=+2386.510037268"
Nov 03 11:09:16.577650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.577449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:16.726689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.726651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:16.863520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.863417    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:16.951539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:16.951509    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:09:17.291985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:17.291942    5817 generic.go:356] "Generic (PLEG): container finished" podID="d7d5df9d-eea5-475d-ac9a-2e046d2ad737" containerID="3669ceda47e5b1ca067b5bb29079c2fa5162359213484d6c6c6f435cb5199c4f" exitCode=0
Nov 03 11:09:17.292490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:17.292187    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4" event={"ID":"d7d5df9d-eea5-475d-ac9a-2e046d2ad737","Type":"ContainerDied","Data":"3669ceda47e5b1ca067b5bb29079c2fa5162359213484d6c6c6f435cb5199c4f"}
Nov 03 11:09:17.407621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:17.406475    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:17.564713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:17.564631    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:17.953447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:17.953244    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:18.134708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.134675    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:18.577041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.576984    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:18.749251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.749222    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-openshift-ca\") pod \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") "
Nov 03 11:09:18.749417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.749346    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-kube-api-access-5zhj7\" (UniqueName: \"kubernetes.io/projected/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-kube-api-access-5zhj7\") pod \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") "
Nov 03 11:09:18.749546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.749533    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-xks-ca\") pod \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\" (UID: \"d7d5df9d-eea5-475d-ac9a-2e046d2ad737\") "
Nov 03 11:09:18.749605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.749558    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-openshift-ca" pod "d7d5df9d-eea5-475d-ac9a-2e046d2ad737" (UID: "d7d5df9d-eea5-475d-ac9a-2e046d2ad737"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:18.750084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.750061    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:18.751556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.751537    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-kube-api-access-5zhj7" pod "d7d5df9d-eea5-475d-ac9a-2e046d2ad737" (UID: "d7d5df9d-eea5-475d-ac9a-2e046d2ad737"). InnerVolumeSpecName "kube-api-access-5zhj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:18.751746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.751726    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-xks-ca" pod "d7d5df9d-eea5-475d-ac9a-2e046d2ad737" (UID: "d7d5df9d-eea5-475d-ac9a-2e046d2ad737"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:09:18.851659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.851517    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:18.851659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.851551    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhj7\" (UniqueName: \"kubernetes.io/projected/d7d5df9d-eea5-475d-ac9a-2e046d2ad737-kube-api-access-5zhj7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:18.977606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:18.975149    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:19.324314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:19.324286    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"
Nov 03 11:09:19.324517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:19.324317    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4" event={"ID":"d7d5df9d-eea5-475d-ac9a-2e046d2ad737","Type":"ContainerDied","Data":"99849e8932563707be77a2c6051c1093fafeb2c642a5ab9305e56a08d7e79481"}
Nov 03 11:09:19.324517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:19.324363    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99849e8932563707be77a2c6051c1093fafeb2c642a5ab9305e56a08d7e79481"
Nov 03 11:09:19.678523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:19.670098    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:20.353594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:20.350896    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:21.094494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:21.094460    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:21.677605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:21.672309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:22.083937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.083909    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"]
Nov 03 11:09:22.084838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.084820    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b0d06c89-d29c-424f-a971-6fddacf7ea29" containerName="revoke-keys"
Nov 03 11:09:22.084838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.084839    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d06c89-d29c-424f-a971-6fddacf7ea29" containerName="revoke-keys"
Nov 03 11:09:22.084965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.084852    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d7d5df9d-eea5-475d-ac9a-2e046d2ad737" containerName="revoke-keys"
Nov 03 11:09:22.084965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.084858    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d5df9d-eea5-475d-ac9a-2e046d2ad737" containerName="revoke-keys"
Nov 03 11:09:22.085057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.085012    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d06c89-d29c-424f-a971-6fddacf7ea29" containerName="revoke-keys"
Nov 03 11:09:22.085057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.085022    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d5df9d-eea5-475d-ac9a-2e046d2ad737" containerName="revoke-keys"
Nov 03 11:09:22.090109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.090083    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.094729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.094707    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:09:22.095157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.094748    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:09:22.096807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.096785    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"]
Nov 03 11:09:22.126505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.126477    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.126698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.126677    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xj5l\" (UniqueName: \"kubernetes.io/projected/c2d7027a-7718-4395-abbe-de6aba1058c1-kube-api-access-2xj5l\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.126790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.126776    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.228525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.228492    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.228740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.228689    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.228910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.228895    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xj5l\" (UniqueName: \"kubernetes.io/projected/c2d7027a-7718-4395-abbe-de6aba1058c1-kube-api-access-2xj5l\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.229385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.229366    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.231516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.231490    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.236480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.236377    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xj5l\" (UniqueName: \"kubernetes.io/projected/c2d7027a-7718-4395-abbe-de6aba1058c1-kube-api-access-2xj5l\") pod \"maas-api-revoke-sub-fb555f4c-rq896\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.383646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.379234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:22.417597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.415181    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:22.462463 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:09:22.462429    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:09:22.730074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.730034    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"]
Nov 03 11:09:22.730216 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:09:22.730041    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d7027a_7718_4395_abbe_de6aba1058c1.slice/crio-b28235b8dd98bd1b273a29ccf89dceb7afbe8ec1a7469a6fa08ca607715f38a2 WatchSource:0}: Error finding container b28235b8dd98bd1b273a29ccf89dceb7afbe8ec1a7469a6fa08ca607715f38a2: Status 404 returned error can't find the container with id b28235b8dd98bd1b273a29ccf89dceb7afbe8ec1a7469a6fa08ca607715f38a2
Nov 03 11:09:22.988219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:22.988123    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:23.443147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:23.443115    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896" event={"ID":"c2d7027a-7718-4395-abbe-de6aba1058c1","Type":"ContainerStarted","Data":"50f510f5fcd187e3a91adf2fb91f4854e2e98c241eaaa7a0783caec4f7578c69"}
Nov 03 11:09:23.443642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:23.443207    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896" event={"ID":"c2d7027a-7718-4395-abbe-de6aba1058c1","Type":"ContainerStarted","Data":"b28235b8dd98bd1b273a29ccf89dceb7afbe8ec1a7469a6fa08ca607715f38a2"}
Nov 03 11:09:23.464432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:23.464383    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896" podStartSLOduration=1.464368609 podStartE2EDuration="1.464368609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:23.461079291 +0000 UTC m=+2393.666661344" watchObservedRunningTime="2026-11-03 11:09:23.464368609 +0000 UTC m=+2393.669950660"
Nov 03 11:09:23.587920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:23.587886    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:24.275260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:24.275181    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:24.455940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:24.455904    5817 generic.go:356] "Generic (PLEG): container finished" podID="c2d7027a-7718-4395-abbe-de6aba1058c1" containerID="50f510f5fcd187e3a91adf2fb91f4854e2e98c241eaaa7a0783caec4f7578c69" exitCode=0
Nov 03 11:09:24.456352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:24.455988    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896" event={"ID":"c2d7027a-7718-4395-abbe-de6aba1058c1","Type":"ContainerDied","Data":"50f510f5fcd187e3a91adf2fb91f4854e2e98c241eaaa7a0783caec4f7578c69"}
Nov 03 11:09:24.938764 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:24.938723    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:25.616373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.611169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:25.716370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.716340    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:25.810031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.809947    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-openshift-ca\") pod \"c2d7027a-7718-4395-abbe-de6aba1058c1\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") "
Nov 03 11:09:25.810179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.810136    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-xks-ca\") pod \"c2d7027a-7718-4395-abbe-de6aba1058c1\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") "
Nov 03 11:09:25.810224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.810178    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d7027a-7718-4395-abbe-de6aba1058c1-kube-api-access-2xj5l\" (UniqueName: \"kubernetes.io/projected/c2d7027a-7718-4395-abbe-de6aba1058c1-kube-api-access-2xj5l\") pod \"c2d7027a-7718-4395-abbe-de6aba1058c1\" (UID: \"c2d7027a-7718-4395-abbe-de6aba1058c1\") "
Nov 03 11:09:25.810382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.810356    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-openshift-ca" pod "c2d7027a-7718-4395-abbe-de6aba1058c1" (UID: "c2d7027a-7718-4395-abbe-de6aba1058c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:25.810955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.810930    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:25.812548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.812523    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d7027a-7718-4395-abbe-de6aba1058c1-kube-api-access-2xj5l" pod "c2d7027a-7718-4395-abbe-de6aba1058c1" (UID: "c2d7027a-7718-4395-abbe-de6aba1058c1"). InnerVolumeSpecName "kube-api-access-2xj5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:25.812895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.812879    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-xks-ca" pod "c2d7027a-7718-4395-abbe-de6aba1058c1" (UID: "c2d7027a-7718-4395-abbe-de6aba1058c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:09:25.912399 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.912373    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d7027a-7718-4395-abbe-de6aba1058c1-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:25.912399 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:25.912398    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xj5l\" (UniqueName: \"kubernetes.io/projected/c2d7027a-7718-4395-abbe-de6aba1058c1-kube-api-access-2xj5l\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:26.250593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:26.250544    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:26.478165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:26.478135    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"
Nov 03 11:09:26.478341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:26.478174    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896" event={"ID":"c2d7027a-7718-4395-abbe-de6aba1058c1","Type":"ContainerDied","Data":"b28235b8dd98bd1b273a29ccf89dceb7afbe8ec1a7469a6fa08ca607715f38a2"}
Nov 03 11:09:26.478341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:26.478257    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28235b8dd98bd1b273a29ccf89dceb7afbe8ec1a7469a6fa08ca607715f38a2"
Nov 03 11:09:26.962418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:26.962386    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:27.548185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:27.548152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:28.189247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:28.189130    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:29.058597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:29.058288    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:29.676284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:29.676248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:30.560142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:30.560092    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:30.981596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:30.980403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:31.954930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:31.952453    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:32.776359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:32.774157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:33.111151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:33.111042    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:33.264230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:33.264198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:33.803915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:33.803885    5817 scope.go:122] "RemoveContainer" containerID="56a3d3f470cff61ebc197f9eca059fa8a32242413a91e7230ca79f9a7a3b476d"
Nov 03 11:09:33.837674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:33.830201    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:33.870591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:33.866769    5817 scope.go:122] "RemoveContainer" containerID="49ef83e27c5762f4e21eb1d6391bc63d16f1a3ed786278d6f2f8c504f87f87fa"
Nov 03 11:09:34.494596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:34.490237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:34.978512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:34.978473    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:35.107728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:35.107679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:35.850593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:35.850015    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:36.533080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:36.533042    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:36.669597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:36.667420    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:37.114642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.110560    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"]
Nov 03 11:09:37.114642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.112196    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c2d7027a-7718-4395-abbe-de6aba1058c1" containerName="revoke-keys"
Nov 03 11:09:37.114642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.112218    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d7027a-7718-4395-abbe-de6aba1058c1" containerName="revoke-keys"
Nov 03 11:09:37.114642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.112378    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d7027a-7718-4395-abbe-de6aba1058c1" containerName="revoke-keys"
Nov 03 11:09:37.129951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.129923    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"]
Nov 03 11:09:37.130276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.130221    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.137052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.137031    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:09:37.138206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.138140    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:09:37.283547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.283516    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.283763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.283591    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59l7n\" (UniqueName: \"kubernetes.io/projected/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-kube-api-access-59l7n\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.283883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.283867    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.384868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.384769    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.384868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.384825    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.385097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.384879    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59l7n\" (UniqueName: \"kubernetes.io/projected/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-kube-api-access-59l7n\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.385425 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.385401    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.387515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.387488    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.393806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.393777    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59l7n\" (UniqueName: \"kubernetes.io/projected/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-kube-api-access-59l7n\") pod \"maas-api-revoke-sub-eb4273ab-zs8t8\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.447831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.447807    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:37.749395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.743994    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"]
Nov 03 11:09:37.794609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.793839    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:37.890111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.890086    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"]
Nov 03 11:09:37.896961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.896936    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:37.917068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.917035    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"]
Nov 03 11:09:37.995408 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.995321    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:37.995625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.995513    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9w6\" (UniqueName: \"kubernetes.io/projected/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-kube-api-access-8m9w6\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:37.995779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:37.995754    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:38.096959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.096928    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:38.097108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.097046    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:38.097185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.097171    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9w6\" (UniqueName: \"kubernetes.io/projected/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-kube-api-access-8m9w6\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:38.097823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.097755    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:38.099667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.099644    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:38.105534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.105513    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9w6\" (UniqueName: \"kubernetes.io/projected/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-kube-api-access-8m9w6\") pod \"maas-api-revoke-sub-a99ec346-5tx5g\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:38.222770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.222742    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:38.480624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.475955    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:38.532861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.532785    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:09:38.545351 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:09:38.545318    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ac1e9c_ddda_4ddc_88f2_d02a114cdb9d.slice/crio-9e48c937ab41f92f53faf9add11942a07c4b31deea196532dfdfc3adfad3156f WatchSource:0}: Error finding container 9e48c937ab41f92f53faf9add11942a07c4b31deea196532dfdfc3adfad3156f: Status 404 returned error can't find the container with id 9e48c937ab41f92f53faf9add11942a07c4b31deea196532dfdfc3adfad3156f
Nov 03 11:09:38.545751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.545724    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"]
Nov 03 11:09:38.609949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.609917    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8" event={"ID":"fd6b8434-ffdb-4383-9d8e-b7a9811c6449","Type":"ContainerStarted","Data":"b5b2188d0c4cbc6c65e2d23600fd7b1eeb525897d8042f4bad6f068a9f1d5923"}
Nov 03 11:09:38.610071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.609975    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8" event={"ID":"fd6b8434-ffdb-4383-9d8e-b7a9811c6449","Type":"ContainerStarted","Data":"958810e6b5b61711e8fb0d862b68d081e22fc8d44554433b3334b983c6bb8bed"}
Nov 03 11:09:38.611508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.611485    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g" event={"ID":"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d","Type":"ContainerStarted","Data":"9e48c937ab41f92f53faf9add11942a07c4b31deea196532dfdfc3adfad3156f"}
Nov 03 11:09:38.630908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:38.630868    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8" podStartSLOduration=1.6308570690000002 podStartE2EDuration="1.630857069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:38.627190081 +0000 UTC m=+2408.832772135" watchObservedRunningTime="2026-11-03 11:09:38.630857069 +0000 UTC m=+2408.836439119"
Nov 03 11:09:39.237638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:39.232006    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:39.680600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:39.677253    5817 generic.go:356] "Generic (PLEG): container finished" podID="fd6b8434-ffdb-4383-9d8e-b7a9811c6449" containerID="b5b2188d0c4cbc6c65e2d23600fd7b1eeb525897d8042f4bad6f068a9f1d5923" exitCode=0
Nov 03 11:09:39.680600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:39.677516    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8" event={"ID":"fd6b8434-ffdb-4383-9d8e-b7a9811c6449","Type":"ContainerDied","Data":"b5b2188d0c4cbc6c65e2d23600fd7b1eeb525897d8042f4bad6f068a9f1d5923"}
Nov 03 11:09:39.686602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:39.682193    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g" event={"ID":"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d","Type":"ContainerStarted","Data":"ad399cbfa530632cb25adf084e695be09ff4c4e351da7aea2622dfd4c23d19bb"}
Nov 03 11:09:40.735595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:40.723518    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 11:09:40.775942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:40.768285    5817 generic.go:356] "Generic (PLEG): container finished" podID="e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d" containerID="ad399cbfa530632cb25adf084e695be09ff4c4e351da7aea2622dfd4c23d19bb" exitCode=0
Nov 03 11:09:40.775942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:40.769162    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g" event={"ID":"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d","Type":"ContainerDied","Data":"ad399cbfa530632cb25adf084e695be09ff4c4e351da7aea2622dfd4c23d19bb"}
Nov 03 11:09:40.847604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:40.840804    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:09:40.857604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:40.857250    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:09:40.964519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:40.962885    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:41.184916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.184889    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:41.220106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.216839    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-xks-ca\") pod \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") "
Nov 03 11:09:41.220106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.216904    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-openshift-ca\") pod \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") "
Nov 03 11:09:41.220106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.217029    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-kube-api-access-59l7n\" (UniqueName: \"kubernetes.io/projected/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-kube-api-access-59l7n\") pod \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\" (UID: \"fd6b8434-ffdb-4383-9d8e-b7a9811c6449\") "
Nov 03 11:09:41.220106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.220032    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-openshift-ca" pod "fd6b8434-ffdb-4383-9d8e-b7a9811c6449" (UID: "fd6b8434-ffdb-4383-9d8e-b7a9811c6449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:41.221239 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.221211    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-kube-api-access-59l7n" pod "fd6b8434-ffdb-4383-9d8e-b7a9811c6449" (UID: "fd6b8434-ffdb-4383-9d8e-b7a9811c6449"). InnerVolumeSpecName "kube-api-access-59l7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:41.223595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.223478    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-xks-ca" pod "fd6b8434-ffdb-4383-9d8e-b7a9811c6449" (UID: "fd6b8434-ffdb-4383-9d8e-b7a9811c6449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:09:41.320133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.320053    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59l7n\" (UniqueName: \"kubernetes.io/projected/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-kube-api-access-59l7n\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:41.320133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.320084    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:41.320133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.320095    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6b8434-ffdb-4383-9d8e-b7a9811c6449-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:41.648645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.643048    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:41.789599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.789514    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"
Nov 03 11:09:41.790131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.789737    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8" event={"ID":"fd6b8434-ffdb-4383-9d8e-b7a9811c6449","Type":"ContainerDied","Data":"958810e6b5b61711e8fb0d862b68d081e22fc8d44554433b3334b983c6bb8bed"}
Nov 03 11:09:41.790131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:41.789786    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="958810e6b5b61711e8fb0d862b68d081e22fc8d44554433b3334b983c6bb8bed"
Nov 03 11:09:42.082359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.082282    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:42.172612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.172344    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-kube-api-access-8m9w6\" (UniqueName: \"kubernetes.io/projected/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-kube-api-access-8m9w6\") pod \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") "
Nov 03 11:09:42.172813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.172709    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-openshift-ca\") pod \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") "
Nov 03 11:09:42.172813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.172750    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-xks-ca\") pod \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\" (UID: \"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d\") "
Nov 03 11:09:42.236648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.222229    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-openshift-ca" pod "e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d" (UID: "e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:42.260593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.258376    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:42.368376 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.344786    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-kube-api-access-8m9w6" pod "e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d" (UID: "e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d"). InnerVolumeSpecName "kube-api-access-8m9w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:42.368376 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.360795    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9w6\" (UniqueName: \"kubernetes.io/projected/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-kube-api-access-8m9w6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:42.459595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.450971    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:42.492981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.459941    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-xks-ca" pod "e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d" (UID: "e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:09:42.492981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.468201    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:42.813648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.810360    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g" event={"ID":"e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d","Type":"ContainerDied","Data":"9e48c937ab41f92f53faf9add11942a07c4b31deea196532dfdfc3adfad3156f"}
Nov 03 11:09:42.813648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.810421    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e48c937ab41f92f53faf9add11942a07c4b31deea196532dfdfc3adfad3156f"
Nov 03 11:09:42.813648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:42.810488    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"
Nov 03 11:09:43.374593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:43.374126    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:43.511594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:43.511326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:43.971789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:43.971403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:44.170213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:44.170138    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"]
Nov 03 11:09:44.229002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:44.228913    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:44.372131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:44.372097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:44.877081 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:44.876790    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:45.276220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:45.276183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:45.908352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:45.908320    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:46.581920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:46.581795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:47.207596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:47.207151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:47.803336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:47.803300    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:48.487237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:48.487200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:48.876442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:48.876359    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:49.050354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.050322    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:49.181654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.177652    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd" podUID="edddee25-fe00-433b-887a-a67597b25099" containerName="payload-processing" containerID="cri-o://50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3" gracePeriod=25
Nov 03 11:09:49.224600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.219155    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:49.674720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.674694    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:09:49.754282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.754255    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-kube-api-access-fvt4j\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-kube-api-access-fvt4j\") pod \"edddee25-fe00-433b-887a-a67597b25099\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") "
Nov 03 11:09:49.754490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.754373    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-routing-overlay-volume\") pod \"edddee25-fe00-433b-887a-a67597b25099\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") "
Nov 03 11:09:49.754490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.754482    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-provider-credentials\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-provider-credentials\") pod \"edddee25-fe00-433b-887a-a67597b25099\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") "
Nov 03 11:09:49.754740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.754717    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-plugins-config-volume\") pod \"edddee25-fe00-433b-887a-a67597b25099\" (UID: \"edddee25-fe00-433b-887a-a67597b25099\") "
Nov 03 11:09:49.754871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.754835    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-routing-overlay-volume" pod "edddee25-fe00-433b-887a-a67597b25099" (UID: "edddee25-fe00-433b-887a-a67597b25099"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:49.755183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.755149    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-plugins-config-volume" pod "edddee25-fe00-433b-887a-a67597b25099" (UID: "edddee25-fe00-433b-887a-a67597b25099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:49.755499 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.755479    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:49.755640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.755502    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edddee25-fe00-433b-887a-a67597b25099-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:49.757823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.757797    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-kube-api-access-fvt4j" pod "edddee25-fe00-433b-887a-a67597b25099" (UID: "edddee25-fe00-433b-887a-a67597b25099"). InnerVolumeSpecName "kube-api-access-fvt4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:49.757927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.757828    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-provider-credentials" pod "edddee25-fe00-433b-887a-a67597b25099" (UID: "edddee25-fe00-433b-887a-a67597b25099"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:49.857527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.857455    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvt4j\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-kube-api-access-fvt4j\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:49.857527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:49.857482    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/edddee25-fe00-433b-887a-a67597b25099-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:50.003506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.003471    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:50.018603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.015549    5817 generic.go:356] "Generic (PLEG): container finished" podID="edddee25-fe00-433b-887a-a67597b25099" containerID="50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3" exitCode=0
Nov 03 11:09:50.018603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.015775    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd" event={"ID":"edddee25-fe00-433b-887a-a67597b25099","Type":"ContainerDied","Data":"50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3"}
Nov 03 11:09:50.018603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.015833    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd" event={"ID":"edddee25-fe00-433b-887a-a67597b25099","Type":"ContainerDied","Data":"15cbea123bd716f17dda747e2f74d11b3ec140a51a07b4407a94b925e78dba28"}
Nov 03 11:09:50.018603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.015866    5817 scope.go:122] "RemoveContainer" containerID="50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3"
Nov 03 11:09:50.018603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.016049    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"
Nov 03 11:09:50.032596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.032559    5817 scope.go:122] "RemoveContainer" containerID="50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3"
Nov 03 11:09:50.033100 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:09:50.033076    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3\": container with ID starting with 50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3 not found: ID does not exist" containerID="50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3"
Nov 03 11:09:50.033258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.033108    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3"} err="failed to get container status \"50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3\": rpc error: code = NotFound desc = could not find container \"50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3\": container with ID starting with 50cac5b1869629c574e891294231f632bb3a65bc2e35d601702e86626d01dfe3 not found: ID does not exist"
Nov 03 11:09:50.038693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.038669    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"]
Nov 03 11:09:50.043884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.043864    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-69bbg8pmd"]
Nov 03 11:09:50.331710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.331672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:50.475959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.475908    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:50.799419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.799387    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"]
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800689    5817 cpu_manager.go:414] "containerMap: removing container" podUID="edddee25-fe00-433b-887a-a67597b25099" containerName="payload-processing"
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800713    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edddee25-fe00-433b-887a-a67597b25099" containerName="payload-processing"
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800731    5817 cpu_manager.go:414] "containerMap: removing container" podUID="fd6b8434-ffdb-4383-9d8e-b7a9811c6449" containerName="revoke-keys"
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800738    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6b8434-ffdb-4383-9d8e-b7a9811c6449" containerName="revoke-keys"
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800756    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d" containerName="revoke-keys"
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800764    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d" containerName="revoke-keys"
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800945    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="edddee25-fe00-433b-887a-a67597b25099" containerName="payload-processing"
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800959    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6b8434-ffdb-4383-9d8e-b7a9811c6449" containerName="revoke-keys"
Nov 03 11:09:50.802279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.800973    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d" containerName="revoke-keys"
Nov 03 11:09:50.806907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.806888    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.829669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.829635    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 11:09:50.829818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.829672    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-abf28fc53a90\""
Nov 03 11:09:50.829818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.829718    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ec531588fd48\""
Nov 03 11:09:50.829818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.829740    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pjcw6\""
Nov 03 11:09:50.829818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.829777    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 11:09:50.829818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.829801    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 11:09:50.830117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.829823    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 11:09:50.833307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.833280    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"]
Nov 03 11:09:50.885853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.885825    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.886140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.886118    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.886247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.886159    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqnv8\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-kube-api-access-gqnv8\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.886315 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.886279    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.987656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.987621    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.987834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.987686    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnv8\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-kube-api-access-gqnv8\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.987834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.987752    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.987981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.987870    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.988552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.988520    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.989099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.989074    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.990443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.990422    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:50.995795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:50.995770    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnv8\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-kube-api-access-gqnv8\") pod \"payload-processing-external-model-855b7646c4-27w2s\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:51.126753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:51.126674    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:51.224015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:51.216261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:51.365802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:51.365760    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:51.473003 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:09:51.472967    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dc1938_0a4d_4383_a745_71acb5138fdc.slice/crio-4ccc59230620dbf889f1ac7f1b15bd34d7fcc29568b7d085e06fb87cdc3eedc4 WatchSource:0}: Error finding container 4ccc59230620dbf889f1ac7f1b15bd34d7fcc29568b7d085e06fb87cdc3eedc4: Status 404 returned error can't find the container with id 4ccc59230620dbf889f1ac7f1b15bd34d7fcc29568b7d085e06fb87cdc3eedc4
Nov 03 11:09:51.474639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:51.474536    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"]
Nov 03 11:09:51.506292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:51.506263    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:51.904938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:51.904867    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edddee25-fe00-433b-887a-a67597b25099" path="/var/lib/kubelet/pods/edddee25-fe00-433b-887a-a67597b25099/volumes"
Nov 03 11:09:52.040840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:52.040806    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s" event={"ID":"36dc1938-0a4d-4383-a745-71acb5138fdc","Type":"ContainerStarted","Data":"d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652"}
Nov 03 11:09:52.041038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:52.040870    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:52.041038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:52.040892    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s" event={"ID":"36dc1938-0a4d-4383-a745-71acb5138fdc","Type":"ContainerStarted","Data":"4ccc59230620dbf889f1ac7f1b15bd34d7fcc29568b7d085e06fb87cdc3eedc4"}
Nov 03 11:09:52.059202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:52.059150    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s" podStartSLOduration=2.059132838 podStartE2EDuration="2.059132838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:52.057794366 +0000 UTC m=+2422.263376434" watchObservedRunningTime="2026-11-03 11:09:52.059132838 +0000 UTC m=+2422.264714890"
Nov 03 11:09:52.091219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:52.091173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:52.415854 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:09:52.415797    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:09:52.809684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:52.782539    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:53.371635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:53.369037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:53.982657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:53.980083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:54.682600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:54.682084    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:55.405668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:55.405625    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:55.973598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:55.972090    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:56.682230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:56.682193    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:57.328817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:57.328780    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:57.480599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:57.477250    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:58.133594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:58.131955    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:58.334653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:58.328858    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:58.782311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:58.772566    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"]
Nov 03 11:09:58.844936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:58.844895    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:09:59.345612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:59.344882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:09:59.878511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:09:59.878471    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:00.031109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:00.031072    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:00.550189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:00.550149    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:00.698183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:00.698140    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:01.191118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:01.191084    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:01.859589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:01.859528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:02.463638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:02.463604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:02.807909 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:02.807825    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:03.087806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:03.087563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:03.769368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:03.769338    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:03.787557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:03.787517    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s" podUID="36dc1938-0a4d-4383-a745-71acb5138fdc" containerName="payload-processing" containerID="cri-o://d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652" gracePeriod=25
Nov 03 11:10:04.081995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.081972    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:10:04.206713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.206682    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-plugins-config-volume\") pod \"36dc1938-0a4d-4383-a745-71acb5138fdc\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") "
Nov 03 11:10:04.207263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.206888    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-provider-credentials\") pod \"36dc1938-0a4d-4383-a745-71acb5138fdc\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") "
Nov 03 11:10:04.207263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.206961    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-kube-api-access-gqnv8\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-kube-api-access-gqnv8\") pod \"36dc1938-0a4d-4383-a745-71acb5138fdc\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") "
Nov 03 11:10:04.207263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.207121    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-plugins-config-volume" pod "36dc1938-0a4d-4383-a745-71acb5138fdc" (UID: "36dc1938-0a4d-4383-a745-71acb5138fdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:04.207263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.207126    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-routing-overlay-volume\") pod \"36dc1938-0a4d-4383-a745-71acb5138fdc\" (UID: \"36dc1938-0a4d-4383-a745-71acb5138fdc\") "
Nov 03 11:10:04.207533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.207507    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-routing-overlay-volume" pod "36dc1938-0a4d-4383-a745-71acb5138fdc" (UID: "36dc1938-0a4d-4383-a745-71acb5138fdc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:04.208042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.208019    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:04.208137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.208045    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dc1938-0a4d-4383-a745-71acb5138fdc-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:04.209496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.209474    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-provider-credentials" pod "36dc1938-0a4d-4383-a745-71acb5138fdc" (UID: "36dc1938-0a4d-4383-a745-71acb5138fdc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:04.209701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.209655    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-kube-api-access-gqnv8" pod "36dc1938-0a4d-4383-a745-71acb5138fdc" (UID: "36dc1938-0a4d-4383-a745-71acb5138fdc"). InnerVolumeSpecName "kube-api-access-gqnv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:04.268902 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.268864    5817 generic.go:356] "Generic (PLEG): container finished" podID="36dc1938-0a4d-4383-a745-71acb5138fdc" containerID="d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652" exitCode=0
Nov 03 11:10:04.269061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.268930    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"
Nov 03 11:10:04.269061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.268944    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s" event={"ID":"36dc1938-0a4d-4383-a745-71acb5138fdc","Type":"ContainerDied","Data":"d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652"}
Nov 03 11:10:04.269061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.269028    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-27w2s" event={"ID":"36dc1938-0a4d-4383-a745-71acb5138fdc","Type":"ContainerDied","Data":"4ccc59230620dbf889f1ac7f1b15bd34d7fcc29568b7d085e06fb87cdc3eedc4"}
Nov 03 11:10:04.269061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.269052    5817 scope.go:122] "RemoveContainer" containerID="d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652"
Nov 03 11:10:04.281151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.281134    5817 scope.go:122] "RemoveContainer" containerID="d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652"
Nov 03 11:10:04.281394 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:10:04.281378    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652\": container with ID starting with d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652 not found: ID does not exist" containerID="d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652"
Nov 03 11:10:04.281446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.281401    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652"} err="failed to get container status \"d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652\": rpc error: code = NotFound desc = could not find container \"d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652\": container with ID starting with d3904780559ff3fba71d9b7d3e9f2b6749b16f9d0c57ba67149fe41be01ba652 not found: ID does not exist"
Nov 03 11:10:04.299607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.299529    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"]
Nov 03 11:10:04.304835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.304812    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-27w2s"]
Nov 03 11:10:04.309938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.309916    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:04.309938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.309937    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqnv8\" (UniqueName: \"kubernetes.io/projected/36dc1938-0a4d-4383-a745-71acb5138fdc-kube-api-access-gqnv8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:04.460601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:04.457418    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:05.061705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.061671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:05.648587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.648538    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"]
Nov 03 11:10:05.649467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.649430    5817 cpu_manager.go:414] "containerMap: removing container" podUID="36dc1938-0a4d-4383-a745-71acb5138fdc" containerName="payload-processing"
Nov 03 11:10:05.649467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.649463    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dc1938-0a4d-4383-a745-71acb5138fdc" containerName="payload-processing"
Nov 03 11:10:05.649738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.649724    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dc1938-0a4d-4383-a745-71acb5138fdc" containerName="payload-processing"
Nov 03 11:10:05.655176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.655155    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.663122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.663096    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-917da2-openshift-default-dockercfg-zw2p5\""
Nov 03 11:10:05.667475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.667446    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"]
Nov 03 11:10:05.768186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.768147    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-envoy\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.768407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.768270    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.768407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.768312    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-credential-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.768407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.768378    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-certs\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.768601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.768482    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aa18f3-4823-4266-9257-f43ae8d69a7d-istiod-ca-cert\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.768682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.768650    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmf6\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-kube-api-access-5mmf6\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.768815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.768798    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-token\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.768905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.768850    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-podinfo\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.769088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.769047    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-data\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.773930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.773786    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:05.870143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870103    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-envoy\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870172    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870209    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-credential-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870252    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-certs\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870413    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aa18f3-4823-4266-9257-f43ae8d69a7d-istiod-ca-cert\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870532    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmf6\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-kube-api-access-5mmf6\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870604    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870607    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-token\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870657    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-podinfo\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870668    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-credential-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870729    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-certs\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.870938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.870840    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-data\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.871177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.871153    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-data\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.871265 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.871184    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aa18f3-4823-4266-9257-f43ae8d69a7d-istiod-ca-cert\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.872624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.872589    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-envoy\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.873284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.873262    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-podinfo\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.878600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.878557    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-token\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.878686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.878612    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmf6\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-kube-api-access-5mmf6\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:05.906088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.906024    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36dc1938-0a4d-4383-a745-71acb5138fdc" path="/var/lib/kubelet/pods/36dc1938-0a4d-4383-a745-71acb5138fdc/volumes"
Nov 03 11:10:05.973049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:05.973025    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:06.268059 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:10:06.268030    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aa18f3_4823_4266_9257_f43ae8d69a7d.slice/crio-937f907da405fce3aae2ef2bc4d80cff7b6d5ed58ec863c77e17955e947d564d WatchSource:0}: Error finding container 937f907da405fce3aae2ef2bc4d80cff7b6d5ed58ec863c77e17955e947d564d: Status 404 returned error can't find the container with id 937f907da405fce3aae2ef2bc4d80cff7b6d5ed58ec863c77e17955e947d564d
Nov 03 11:10:06.268654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:06.268529    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"]
Nov 03 11:10:06.271234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:06.271199    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:06.271351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:06.271264    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:06.271351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:06.271298    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:06.291511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:06.291476    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" event={"ID":"04aa18f3-4823-4266-9257-f43ae8d69a7d","Type":"ContainerStarted","Data":"937f907da405fce3aae2ef2bc4d80cff7b6d5ed58ec863c77e17955e947d564d"}
Nov 03 11:10:06.403194 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:06.403162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:06.541654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:06.538321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:06.690633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:06.690543    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:07.089647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:07.080864    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:07.327557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:07.327510    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" event={"ID":"04aa18f3-4823-4266-9257-f43ae8d69a7d","Type":"ContainerStarted","Data":"67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285"}
Nov 03 11:10:07.352612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:07.352494    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" podStartSLOduration=2.352480458 podStartE2EDuration="2.352480458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:07.347531201 +0000 UTC m=+2437.553113264" watchObservedRunningTime="2026-11-03 11:10:07.352480458 +0000 UTC m=+2437.558062509"
Nov 03 11:10:07.863127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:07.863089    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:07.973768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:07.973738    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:07.975203 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:07.975179    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 11:10:07.975318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:07.975251    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 11:10:08.488608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:08.485473    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:08.974229 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:08.974196    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 11:10:08.974657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:08.974257    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 11:10:09.240381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:09.240305    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:09.420652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:09.416996    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:09.974149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:09.974120    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 11:10:09.974387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:09.974363    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 11:10:10.355281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:10.355200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:10.500607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:10.492905    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:10.984600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:10.984051    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 11:10:10.984600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:10.984116    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 11:10:11.006607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:10.985314    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"]
Nov 03 11:10:11.020694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.013816    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.052635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.052184    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pjcw6\""
Nov 03 11:10:11.052635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.052246    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-abf28fc53a90\""
Nov 03 11:10:11.052635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.052279    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ec531588fd48\""
Nov 03 11:10:11.054285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.053274    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fd1ec75a88bc\""
Nov 03 11:10:11.054285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.053346    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 11:10:11.054285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.053401    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 11:10:11.054285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.053484    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 11:10:11.054285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.053557    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 11:10:11.054285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.054252    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"]
Nov 03 11:10:11.118305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.112752    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-provider-credentials\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.118305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.117690    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-routing-overlay-volume\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.118305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.117841    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-plugins-config-volume\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.118305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.118058    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dxj8\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-kube-api-access-8dxj8\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.220945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.219563    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-plugins-config-volume\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.220945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.219774    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dxj8\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-kube-api-access-8dxj8\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.220945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.219883    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-provider-credentials\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.220945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.219918    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-routing-overlay-volume\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.220945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.220563    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-routing-overlay-volume\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.220945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.220812    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-plugins-config-volume\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.223780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.223751    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-provider-credentials\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.228188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.228168    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dxj8\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-kube-api-access-8dxj8\") pod \"payload-processing-external-model-8444774d4b-b2wgn\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.360494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.360407    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:11.662677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:11.662594    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:12.069667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.057208    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"]
Nov 03 11:10:12.118231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.062924    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 11:10:12.143600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.118479    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 11:10:12.158172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.158135    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"]
Nov 03 11:10:12.195767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.189917    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"]
Nov 03 11:10:12.195767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.190040    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287455    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723334-88fc-4f50-93be-7c4af48df73f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287509    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723334-88fc-4f50-93be-7c4af48df73f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287617    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287713    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287814    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287840    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287876    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287908    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.296620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.287935    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqzw4\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-kube-api-access-vqzw4\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.385671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.385636    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn" event={"ID":"2412251d-c2bd-4edd-8c66-3c43258830f1","Type":"ContainerStarted","Data":"b80218b08f15e9f3188c479036461115ceed71293104a9f8ffe51754f943d38d"}
Nov 03 11:10:12.385803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.385700    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn" event={"ID":"2412251d-c2bd-4edd-8c66-3c43258830f1","Type":"ContainerStarted","Data":"08f158d1b70688a121aa55ff07bd1100f1544b83b168d3f8c7c4133eb19c6125"}
Nov 03 11:10:12.385803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.385730    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:12.392624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.392475    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723334-88fc-4f50-93be-7c4af48df73f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.392624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.392537    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723334-88fc-4f50-93be-7c4af48df73f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.392624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.392615    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.407359    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723334-88fc-4f50-93be-7c4af48df73f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.407755    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.407877    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.407908    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.407951    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.407996    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.408027    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqzw4\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-kube-api-access-vqzw4\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.408983    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.409208    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.410632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.409424    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.422594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.421188    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.426136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.425833    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.426378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.426349    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723334-88fc-4f50-93be-7c4af48df73f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.426794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.426749    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn" podStartSLOduration=2.426732038 podStartE2EDuration="2.426732038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:12.410888771 +0000 UTC m=+2442.616470822" watchObservedRunningTime="2026-11-03 11:10:12.426732038 +0000 UTC m=+2442.632314090"
Nov 03 11:10:12.428711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.428690    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqzw4\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-kube-api-access-vqzw4\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.430052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.430031    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vkhvr\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.481023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.480992    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:12.572711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.572677    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:12.640470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.640431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:12.809179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.809146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:12.991061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.981651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:12.991061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.983170    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 11:10:12.991061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:12.983226    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 11:10:13.249739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:13.238726    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:13.440616 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:10:13.438340    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6723334_88fc_4f50_93be_7c4af48df73f.slice/crio-7fef6aeb3584d87bec21b78258af85dab85885a3b83595df363c6c95883f453c WatchSource:0}: Error finding container 7fef6aeb3584d87bec21b78258af85dab85885a3b83595df363c6c95883f453c: Status 404 returned error can't find the container with id 7fef6aeb3584d87bec21b78258af85dab85885a3b83595df363c6c95883f453c
Nov 03 11:10:13.440616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:13.439337    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"]
Nov 03 11:10:13.443885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:13.443848    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.444091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:13.443930    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.444091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:13.443968    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.635597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:13.633845    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:13.801870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:13.801641    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:13.933270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:13.933214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:14.372026 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.371995    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:14.372617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.372596    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:14.374080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.374057    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:10:14.453052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.449200    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" event={"ID":"e6723334-88fc-4f50-93be-7c4af48df73f","Type":"ContainerStarted","Data":"8286f6563e8d1e164681030ae31a725c9653a2a3467a670beec7856f03abd222"}
Nov 03 11:10:14.453052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.449266    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" event={"ID":"e6723334-88fc-4f50-93be-7c4af48df73f","Type":"ContainerStarted","Data":"7fef6aeb3584d87bec21b78258af85dab85885a3b83595df363c6c95883f453c"}
Nov 03 11:10:14.481512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.479985    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podStartSLOduration=2.479954544 podStartE2EDuration="2.479954544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:14.47235028 +0000 UTC m=+2444.677932334" watchObservedRunningTime="2026-11-03 11:10:14.479954544 +0000 UTC m=+2444.685536596"
Nov 03 11:10:14.498976 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.498942    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:14.572830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.572797    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:14.574232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.574206    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:14.574377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:14.574260    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:15.573407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:15.573369    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:15.573856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:15.573424    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:16.361316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:16.361278    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:16.573006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:16.572968    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:16.573251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:16.573020    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:17.573971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:17.573933    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:17.574766 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:17.574734    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:17.962619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:17.953435    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:18.173936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:18.173890    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 03 11:10:18.242781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:18.242546    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:18.342715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:18.342668    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:18.573306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:18.573216    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:18.573306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:18.573283    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:18.962599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:18.962532    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:19.171963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:19.171935    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"]
Nov 03 11:10:19.172187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:19.172163    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy" containerID="cri-o://b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03" gracePeriod=30
Nov 03 11:10:19.581957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:19.581916    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:19.614699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:19.614652    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:20.575143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:20.575109    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:20.575810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:20.575761    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:21.454614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:21.444707    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:21.574419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:21.573971    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:21.574419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:21.574030    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:21.773967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:21.773890    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:21.941299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:21.941267    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:22.032651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.032354    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"]
Nov 03 11:10:22.048849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.048823    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:22.594597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.586278    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:22.594597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.586375    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:22.620454 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:10:22.620410    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:10:22.905091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.904997    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"]
Nov 03 11:10:22.910470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.910447    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:22.922964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.922936    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:10:22.923629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.923606    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:10:22.924176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.924157    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"]
Nov 03 11:10:22.976536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.976510    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:22.976696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.976546    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:22.976749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:22.976728    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrtt\" (UniqueName: \"kubernetes.io/projected/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-kube-api-access-lcrtt\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:23.078424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.078388    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrtt\" (UniqueName: \"kubernetes.io/projected/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-kube-api-access-lcrtt\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:23.078705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.078689    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:23.078784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.078716    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:23.079428 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.079382    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:23.082120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.082090    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:23.087737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.087712    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrtt\" (UniqueName: \"kubernetes.io/projected/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-kube-api-access-lcrtt\") pod \"maas-api-revoke-sub-70808cef-swvxt\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:23.242804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.242768    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:23.565632 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:10:23.565602    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2ab2a2_8efa_48bf_b853_0984cc47acdc.slice/crio-3190db2eaa0e56b4f7828e1257ac11ebf0951b87d86a6d5fe96f52aa18141cfb WatchSource:0}: Error finding container 3190db2eaa0e56b4f7828e1257ac11ebf0951b87d86a6d5fe96f52aa18141cfb: Status 404 returned error can't find the container with id 3190db2eaa0e56b4f7828e1257ac11ebf0951b87d86a6d5fe96f52aa18141cfb
Nov 03 11:10:23.565816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.565790    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"]
Nov 03 11:10:23.573989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.573964    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 11:10:23.574099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.574025    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 11:10:23.697849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.697812    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:23.832343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:23.828332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:24.183659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.181453    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt" event={"ID":"ff2ab2a2-8efa-48bf-b853-0984cc47acdc","Type":"ContainerStarted","Data":"3190db2eaa0e56b4f7828e1257ac11ebf0951b87d86a6d5fe96f52aa18141cfb"}
Nov 03 11:10:24.565779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.565743    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:10:24.667468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667435    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-data\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.667665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667556    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-credential-socket\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.667713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667661    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-envoy\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.667770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667711    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-socket\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.667838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667794    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-kube-api-access-t55zc\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-kube-api-access-t55zc\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.667838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667812    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-data" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:10:24.667955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667837    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-certs\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.667955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667841    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-credential-socket" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:10:24.667955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667883    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-istio-token\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-istio-token\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.667955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.667913    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7948a568-6219-4907-a796-f32e9c6c9bac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7948a568-6219-4907-a796-f32e9c6c9bac-istiod-ca-cert\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.668218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.668000    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7948a568-6219-4907-a796-f32e9c6c9bac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7948a568-6219-4907-a796-f32e9c6c9bac-istio-podinfo\") pod \"7948a568-6219-4907-a796-f32e9c6c9bac\" (UID: \"7948a568-6219-4907-a796-f32e9c6c9bac\") "
Nov 03 11:10:24.668218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.668065    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-socket" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:10:24.668403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.668363    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-certs" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:10:24.668483 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.668399    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7948a568-6219-4907-a796-f32e9c6c9bac-istiod-ca-cert" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:24.669002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.668955    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:24.669002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.668986    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:24.669172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.669006    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:24.669172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.669122    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:24.669172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.669136    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7948a568-6219-4907-a796-f32e9c6c9bac-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:24.670629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.670566    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-envoy" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:10:24.670890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.670801    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-kube-api-access-t55zc" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "kube-api-access-t55zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:24.670890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.670830    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-istio-token" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:24.671042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.671024    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7948a568-6219-4907-a796-f32e9c6c9bac-istio-podinfo" pod "7948a568-6219-4907-a796-f32e9c6c9bac" (UID: "7948a568-6219-4907-a796-f32e9c6c9bac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:10:24.771151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.771059    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7948a568-6219-4907-a796-f32e9c6c9bac-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:24.771151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.771091    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55zc\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-kube-api-access-t55zc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:24.771151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.771101    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7948a568-6219-4907-a796-f32e9c6c9bac-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:24.771151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:24.771109    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7948a568-6219-4907-a796-f32e9c6c9bac-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:25.225164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.225126    5817 generic.go:356] "Generic (PLEG): container finished" podID="ff2ab2a2-8efa-48bf-b853-0984cc47acdc" containerID="92e20b582c71079870486fe65350fa751dca6a42ea87cb88a1c9fbe186556dec" exitCode=0
Nov 03 11:10:25.225709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.225244    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt" event={"ID":"ff2ab2a2-8efa-48bf-b853-0984cc47acdc","Type":"ContainerDied","Data":"92e20b582c71079870486fe65350fa751dca6a42ea87cb88a1c9fbe186556dec"}
Nov 03 11:10:25.228073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.228040    5817 generic.go:356] "Generic (PLEG): container finished" podID="7948a568-6219-4907-a796-f32e9c6c9bac" containerID="b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03" exitCode=0
Nov 03 11:10:25.228271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.228253    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"
Nov 03 11:10:25.228338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.228283    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" event={"ID":"7948a568-6219-4907-a796-f32e9c6c9bac","Type":"ContainerDied","Data":"b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03"}
Nov 03 11:10:25.228338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.228331    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2" event={"ID":"7948a568-6219-4907-a796-f32e9c6c9bac","Type":"ContainerDied","Data":"bf37b05ee08335313d55081d78ced75a697a14eff07c77b01989418c822f04ec"}
Nov 03 11:10:25.228458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.228364    5817 scope.go:122] "RemoveContainer" containerID="b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03"
Nov 03 11:10:25.237126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.237098    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:25.271716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.271631    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"]
Nov 03 11:10:25.275736 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.275704    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-f49f2"]
Nov 03 11:10:25.307981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.307794    5817 scope.go:122] "RemoveContainer" containerID="b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03"
Nov 03 11:10:25.316901 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:10:25.316801    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03\": container with ID starting with b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03 not found: ID does not exist" containerID="b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03"
Nov 03 11:10:25.316901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.316848    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03"} err="failed to get container status \"b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03\": rpc error: code = NotFound desc = could not find container \"b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03\": container with ID starting with b9159478cd8dc19ba33a07fa0edc0a49ed60be6ae1ee80a345fdb2f8b267ac03 not found: ID does not exist"
Nov 03 11:10:25.388607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.384972    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-449q9"]
Nov 03 11:10:25.388607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.386412    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy"
Nov 03 11:10:25.388607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.386436    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy"
Nov 03 11:10:25.388607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.386662    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" containerName="istio-proxy"
Nov 03 11:10:25.400595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.399182    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.411593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.411187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:25.438599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.437623    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-449q9"]
Nov 03 11:10:25.497978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.493452    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-png4x\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-kube-api-access-png4x\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.497978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.493547    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.497978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.493599    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.497978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.493953    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.588654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.586680    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:10:25.588654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.586748    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:25.619598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.619106    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.619598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.619163    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.619598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.619471    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.619934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.619624    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-png4x\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-kube-api-access-png4x\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.638470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.638420    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.640926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.640805    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.762593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.748726    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-png4x\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-kube-api-access-png4x\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.762593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.754189    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-449q9\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:25.913324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:25.913239    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7948a568-6219-4907-a796-f32e9c6c9bac" path="/var/lib/kubelet/pods/7948a568-6219-4907-a796-f32e9c6c9bac/volumes"
Nov 03 11:10:26.035775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.035744    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:26.193594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.192244    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:26.341163 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.341073    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-449q9"]
Nov 03 11:10:26.536944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.536916    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:26.588013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.587896    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:10:26.588013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.587959    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:26.588261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.588228    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-xks-ca\") pod \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") "
Nov 03 11:10:26.589226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.588422    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-kube-api-access-lcrtt\" (UniqueName: \"kubernetes.io/projected/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-kube-api-access-lcrtt\") pod \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") "
Nov 03 11:10:26.589226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.588618    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-openshift-ca\") pod \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\" (UID: \"ff2ab2a2-8efa-48bf-b853-0984cc47acdc\") "
Nov 03 11:10:26.589226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.589108    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-openshift-ca" pod "ff2ab2a2-8efa-48bf-b853-0984cc47acdc" (UID: "ff2ab2a2-8efa-48bf-b853-0984cc47acdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:26.589774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.589492    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:26.591503 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.591472    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-kube-api-access-lcrtt" pod "ff2ab2a2-8efa-48bf-b853-0984cc47acdc" (UID: "ff2ab2a2-8efa-48bf-b853-0984cc47acdc"). InnerVolumeSpecName "kube-api-access-lcrtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:26.591778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.591754    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-xks-ca" pod "ff2ab2a2-8efa-48bf-b853-0984cc47acdc" (UID: "ff2ab2a2-8efa-48bf-b853-0984cc47acdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:10:26.691446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.691409    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:26.691614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:26.691453    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrtt\" (UniqueName: \"kubernetes.io/projected/ff2ab2a2-8efa-48bf-b853-0984cc47acdc-kube-api-access-lcrtt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:27.033952 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.033891    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn" podUID="2412251d-c2bd-4edd-8c66-3c43258830f1" containerName="payload-processing" containerID="cri-o://b80218b08f15e9f3188c479036461115ceed71293104a9f8ffe51754f943d38d" gracePeriod=25
Nov 03 11:10:27.367415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.367273    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:27.399547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.399509    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9" event={"ID":"310d71bb-1549-407b-8ce4-0b0383ca4429","Type":"ContainerStarted","Data":"da2e511a02d1abf3ac855517d978c9c09541b64a190f42b745270b807fc3f525"}
Nov 03 11:10:27.399831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.399807    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9" event={"ID":"310d71bb-1549-407b-8ce4-0b0383ca4429","Type":"ContainerStarted","Data":"c7539211cd18058704b8d79bc82bb0ce28496776631900ab45dfcd4134cc4a95"}
Nov 03 11:10:27.401183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.401123    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:27.408220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.408039    5817 generic.go:356] "Generic (PLEG): container finished" podID="2412251d-c2bd-4edd-8c66-3c43258830f1" containerID="b80218b08f15e9f3188c479036461115ceed71293104a9f8ffe51754f943d38d" exitCode=0
Nov 03 11:10:27.408220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.408128    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn" event={"ID":"2412251d-c2bd-4edd-8c66-3c43258830f1","Type":"ContainerDied","Data":"b80218b08f15e9f3188c479036461115ceed71293104a9f8ffe51754f943d38d"}
Nov 03 11:10:27.410863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.410540    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt" event={"ID":"ff2ab2a2-8efa-48bf-b853-0984cc47acdc","Type":"ContainerDied","Data":"3190db2eaa0e56b4f7828e1257ac11ebf0951b87d86a6d5fe96f52aa18141cfb"}
Nov 03 11:10:27.410863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.410601    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3190db2eaa0e56b4f7828e1257ac11ebf0951b87d86a6d5fe96f52aa18141cfb"
Nov 03 11:10:27.410863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.410678    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"
Nov 03 11:10:27.423629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.422720    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9" podStartSLOduration=2.422703322 podStartE2EDuration="2.422703322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:27.420825423 +0000 UTC m=+2457.626407477" watchObservedRunningTime="2026-11-03 11:10:27.422703322 +0000 UTC m=+2457.628285374"
Nov 03 11:10:27.526238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.526210    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:27.563623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.561906    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-provider-credentials\") pod \"2412251d-c2bd-4edd-8c66-3c43258830f1\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") "
Nov 03 11:10:27.563623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.561992    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-plugins-config-volume\") pod \"2412251d-c2bd-4edd-8c66-3c43258830f1\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") "
Nov 03 11:10:27.563623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.562140    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-routing-overlay-volume\") pod \"2412251d-c2bd-4edd-8c66-3c43258830f1\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") "
Nov 03 11:10:27.563623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.562263    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-kube-api-access-8dxj8\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-kube-api-access-8dxj8\") pod \"2412251d-c2bd-4edd-8c66-3c43258830f1\" (UID: \"2412251d-c2bd-4edd-8c66-3c43258830f1\") "
Nov 03 11:10:27.563623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.562732    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-plugins-config-volume" pod "2412251d-c2bd-4edd-8c66-3c43258830f1" (UID: "2412251d-c2bd-4edd-8c66-3c43258830f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:27.563623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.563000    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:27.563623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.563146    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-routing-overlay-volume" pod "2412251d-c2bd-4edd-8c66-3c43258830f1" (UID: "2412251d-c2bd-4edd-8c66-3c43258830f1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:27.564915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.564883    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-provider-credentials" pod "2412251d-c2bd-4edd-8c66-3c43258830f1" (UID: "2412251d-c2bd-4edd-8c66-3c43258830f1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:27.565562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.565534    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-kube-api-access-8dxj8" pod "2412251d-c2bd-4edd-8c66-3c43258830f1" (UID: "2412251d-c2bd-4edd-8c66-3c43258830f1"). InnerVolumeSpecName "kube-api-access-8dxj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:27.572807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.572784    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:27.629813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.629724    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:10:27.630086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.630057    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:27.665386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.665352    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2412251d-c2bd-4edd-8c66-3c43258830f1-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:27.665386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.665387    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dxj8\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-kube-api-access-8dxj8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:27.665757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:27.665400    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2412251d-c2bd-4edd-8c66-3c43258830f1-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:28.114452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.105290    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:28.428095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.428053    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn" event={"ID":"2412251d-c2bd-4edd-8c66-3c43258830f1","Type":"ContainerDied","Data":"08f158d1b70688a121aa55ff07bd1100f1544b83b168d3f8c7c4133eb19c6125"}
Nov 03 11:10:28.428740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.428706    5817 scope.go:122] "RemoveContainer" containerID="b80218b08f15e9f3188c479036461115ceed71293104a9f8ffe51754f943d38d"
Nov 03 11:10:28.428992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.428093    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"
Nov 03 11:10:28.459305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.459271    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"]
Nov 03 11:10:28.465702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.465675    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-b2wgn"]
Nov 03 11:10:28.630798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.630761    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:10:28.631105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.631068    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:28.710919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.709382    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:28.714844 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.714489    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:10:28.934599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:28.924160    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:29.790248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:29.790213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:29.907496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:29.907465    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2412251d-c2bd-4edd-8c66-3c43258830f1" path="/var/lib/kubelet/pods/2412251d-c2bd-4edd-8c66-3c43258830f1/volumes"
Nov 03 11:10:30.775465 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:30.775431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:31.623602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:31.623121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:32.624506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:32.624468    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:33.375593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:33.374892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:33.968113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:33.968076    5817 scope.go:122] "RemoveContainer" containerID="fbe1c51232e0b737e2e511002e756f51e5dd183056a6b3d6b32fe3e7670104fe"
Nov 03 11:10:34.099995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:34.097030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:34.700591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:34.699542    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:35.557122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:35.555843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:36.050963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.050925    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:36.221219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.220619    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-449q9"]
Nov 03 11:10:36.221219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.221198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:36.286807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.286773    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"]
Nov 03 11:10:36.309660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.288451    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2412251d-c2bd-4edd-8c66-3c43258830f1" containerName="payload-processing"
Nov 03 11:10:36.309949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.309919    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2412251d-c2bd-4edd-8c66-3c43258830f1" containerName="payload-processing"
Nov 03 11:10:36.310225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.310206    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ff2ab2a2-8efa-48bf-b853-0984cc47acdc" containerName="revoke-keys"
Nov 03 11:10:36.310347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.310333    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2ab2a2-8efa-48bf-b853-0984cc47acdc" containerName="revoke-keys"
Nov 03 11:10:36.333601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.315927    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2ab2a2-8efa-48bf-b853-0984cc47acdc" containerName="revoke-keys"
Nov 03 11:10:36.333934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.333909    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2412251d-c2bd-4edd-8c66-3c43258830f1" containerName="payload-processing"
Nov 03 11:10:36.359372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.359348    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:36.359918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.359900    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.394707 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.394677    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"]
Nov 03 11:10:36.466287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.466251    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-provider-credentials\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.466462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.466302    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-plugins-config-volume\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.466462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.466379    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qh86\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-kube-api-access-6qh86\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.466462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.466440    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-routing-overlay-volume\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.571678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.569116    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-provider-credentials\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.571678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.569186    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-plugins-config-volume\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.571678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.569215    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qh86\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-kube-api-access-6qh86\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.571678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.569284    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-routing-overlay-volume\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.571678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.570289    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-routing-overlay-volume\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.590629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.585123    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-plugins-config-volume\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.610593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.596127    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-provider-credentials\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.618762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.618734    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qh86\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-kube-api-access-6qh86\") pod \"payload-processing-external-model-68fb489997-rw4h2\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") " pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:36.693636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:36.693603    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:37.076598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:37.076118    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:37.125479 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:10:37.125441    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10493144_b203_457e_8e5e_b8419e8fe53b.slice/crio-6c4401424ab499bcf6c20c741835011d2c0be4189c8f401b66593fdacc2229ab WatchSource:0}: Error finding container 6c4401424ab499bcf6c20c741835011d2c0be4189c8f401b66593fdacc2229ab: Status 404 returned error can't find the container with id 6c4401424ab499bcf6c20c741835011d2c0be4189c8f401b66593fdacc2229ab
Nov 03 11:10:37.126968 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:37.126942    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"]
Nov 03 11:10:37.633657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:37.633626    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2" event={"ID":"10493144-b203-457e-8e5e-b8419e8fe53b","Type":"ContainerStarted","Data":"544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737"}
Nov 03 11:10:37.634084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:37.633680    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2" event={"ID":"10493144-b203-457e-8e5e-b8419e8fe53b","Type":"ContainerStarted","Data":"6c4401424ab499bcf6c20c741835011d2c0be4189c8f401b66593fdacc2229ab"}
Nov 03 11:10:37.634084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:37.633881    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:37.694389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:37.694331    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2" podStartSLOduration=1.694316468 podStartE2EDuration="1.694316468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:37.688446533 +0000 UTC m=+2467.894028583" watchObservedRunningTime="2026-11-03 11:10:37.694316468 +0000 UTC m=+2467.899898525"
Nov 03 11:10:37.832189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:37.832155    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:38.004649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:38.004615    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:38.182597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:38.173920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:38.337661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:38.335075    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:38.475209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:38.475169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:38.692241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:38.685832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:38.964015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:38.963927    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:39.160959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.160914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:39.307643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.307088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:39.636655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.635686    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"]
Nov 03 11:10:39.643698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.643675    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.654628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.654534    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"kube-root-ca.crt\""
Nov 03 11:10:39.654790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.654682    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"payload-processing-external-model-e2e--ac720701-dockercfg-hq9vb\""
Nov 03 11:10:39.654868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.654796    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-35936b\""
Nov 03 11:10:39.654928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.654893    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"agc-cred-a1416845a683\""
Nov 03 11:10:39.655013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.654999    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"openshift-service-ca.crt\""
Nov 03 11:10:39.655087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.655063    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"routing-overlay\""
Nov 03 11:10:39.672869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.668135    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"]
Nov 03 11:10:39.672869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.668708    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:39.828599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.805204    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26w65\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-kube-api-access-26w65\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.828599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.805315    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.828599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.805371    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.828599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.805435    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.916200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.916094    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.943956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.939810    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.943956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.941126    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.943956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.941361    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.943956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.942339    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.943956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.943904    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.952890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.952794    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26w65\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-kube-api-access-26w65\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:39.980201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.980166    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:40.006500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:39.999263    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26w65\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-kube-api-access-26w65\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:40.271381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:40.271348    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:40.272690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:40.272658    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:40.696645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:40.618024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:41.090599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.075725    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"]
Nov 03 11:10:41.144597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.090942    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:41.225520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.225313    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9" podUID="310d71bb-1549-407b-8ce4-0b0383ca4429" containerName="payload-processing" containerID="cri-o://da2e511a02d1abf3ac855517d978c9c09541b64a190f42b745270b807fc3f525" gracePeriod=25
Nov 03 11:10:41.335647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.318147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:41.480456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.480402    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:41.508888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.506959    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"]
Nov 03 11:10:41.760745 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.760693    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:41.835712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.792886    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:41.835712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.793155    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:41.871172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.871132    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 11:10:41.911684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.899538    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:41.929636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.921722    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:41.929636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.922640    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 11:10:41.980426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:41.980383    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:42.139162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.108841    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.139162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.119086    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"openshift-service-ca.crt\""
Nov 03 11:10:42.139162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.119379    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"agc-cred-ea1161eb6e3d\""
Nov 03 11:10:42.139162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.120295    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d9f2ce\""
Nov 03 11:10:42.139162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.120420    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"routing-overlay\""
Nov 03 11:10:42.139162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.120613    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"kube-root-ca.crt\""
Nov 03 11:10:42.145598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.145479    5817 generic.go:356] "Generic (PLEG): container finished" podID="310d71bb-1549-407b-8ce4-0b0383ca4429" containerID="da2e511a02d1abf3ac855517d978c9c09541b64a190f42b745270b807fc3f525" exitCode=0
Nov 03 11:10:42.180244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.180201    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:42.196684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.187982    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj" event={"ID":"05f96069-99fe-4409-9415-d674a1da3ed1","Type":"ContainerStarted","Data":"dfa3eb18a4a6cacf54c3ae71b4a8b538e5803479d2332aaa54c4415529caac16"}
Nov 03 11:10:42.196684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.188057    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"]
Nov 03 11:10:42.196684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.188091    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9" event={"ID":"310d71bb-1549-407b-8ce4-0b0383ca4429","Type":"ContainerDied","Data":"da2e511a02d1abf3ac855517d978c9c09541b64a190f42b745270b807fc3f525"}
Nov 03 11:10:42.196684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.188121    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"]
Nov 03 11:10:42.196684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.194487    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.216598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.216018    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"routing-overlay\""
Nov 03 11:10:42.216598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.216104    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"agc-cred-b8c3e972a39f\""
Nov 03 11:10:42.216598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.216141    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"agc-cred-6096dd351669\""
Nov 03 11:10:42.216598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.216138    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"payload-processing-external-model-e2e--7ca7b2d5-dockercfg-cvxnx\""
Nov 03 11:10:42.216598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.216189    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"openshift-service-ca.crt\""
Nov 03 11:10:42.216598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.216265    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"kube-root-ca.crt\""
Nov 03 11:10:42.216598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.216285    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5711cf\""
Nov 03 11:10:42.247255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.247221    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"]
Nov 03 11:10:42.247724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.247692    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": context deadline exceeded"
Nov 03 11:10:42.265786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.263647    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.265786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.263713    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfq52\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-kube-api-access-vfq52\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.265786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.264107    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.265786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.264382    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.303386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.303352    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"]
Nov 03 11:10:42.320083 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.320060    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"]
Nov 03 11:10:42.320283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.320268    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.324513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.324490    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"agc-cred-5ef6a65b55a2\""
Nov 03 11:10:42.365554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.365526    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.365716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.365588    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.365716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.365623    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.365716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.365658    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.365716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.365686    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d667\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-kube-api-access-4d667\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.365988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.365728    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.365988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.365756    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfq52\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-kube-api-access-vfq52\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.365988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.365896    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.366977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.366931    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.367498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.367475    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.368906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.368821    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.375272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.375247    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfq52\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-kube-api-access-vfq52\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.413703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.413680    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:42.466136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.466089    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:42.467268 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.467238    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.467407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.467290    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.467407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.467341    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.467407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.467374    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqp2\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-kube-api-access-hfqp2\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.467655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.467420    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d667\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-kube-api-access-4d667\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.467655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.467491    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.467655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.467529    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.467856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.467720    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.468471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.468441    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.468764 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.468740    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.471828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.471772    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.479530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.479496    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d667\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-kube-api-access-4d667\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.569702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.569280    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-provider-credentials\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-provider-credentials\") pod \"310d71bb-1549-407b-8ce4-0b0383ca4429\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") "
Nov 03 11:10:42.569921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.569787    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-kube-api-access-png4x\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-kube-api-access-png4x\") pod \"310d71bb-1549-407b-8ce4-0b0383ca4429\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") "
Nov 03 11:10:42.569921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.569838    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-plugins-config-volume\") pod \"310d71bb-1549-407b-8ce4-0b0383ca4429\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") "
Nov 03 11:10:42.569921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.569908    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-routing-overlay-volume\") pod \"310d71bb-1549-407b-8ce4-0b0383ca4429\" (UID: \"310d71bb-1549-407b-8ce4-0b0383ca4429\") "
Nov 03 11:10:42.570628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.570604    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqp2\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-kube-api-access-hfqp2\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.570756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.570716    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.570816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.570758    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.570940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.570923    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.573019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.572385    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-plugins-config-volume" pod "310d71bb-1549-407b-8ce4-0b0383ca4429" (UID: "310d71bb-1549-407b-8ce4-0b0383ca4429"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:42.573019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.572837    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-provider-credentials" pod "310d71bb-1549-407b-8ce4-0b0383ca4429" (UID: "310d71bb-1549-407b-8ce4-0b0383ca4429"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:42.573019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.572994    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-routing-overlay-volume" pod "310d71bb-1549-407b-8ce4-0b0383ca4429" (UID: "310d71bb-1549-407b-8ce4-0b0383ca4429"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:42.573429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.573238    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.573429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.573389    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.575161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.575138    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.580702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.580679    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqp2\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-kube-api-access-hfqp2\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.583644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.583437    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:42.584988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.584964    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-kube-api-access-png4x" pod "310d71bb-1549-407b-8ce4-0b0383ca4429" (UID: "310d71bb-1549-407b-8ce4-0b0383ca4429"). InnerVolumeSpecName "kube-api-access-png4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:42.658753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.658724    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:42.673680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.673650    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-png4x\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-kube-api-access-png4x\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:42.673780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.673684    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:42.673780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.673704    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/310d71bb-1549-407b-8ce4-0b0383ca4429-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:42.673780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.673717    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/310d71bb-1549-407b-8ce4-0b0383ca4429-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:42.768179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.768151    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"]
Nov 03 11:10:42.768468 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:10:42.768444    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d76f4dd_9618_4ead_8cd2_b999526c1454.slice/crio-8ade60c1a422914833f60bd84bfbdeee855f59258265d6fba54302b24cc70095 WatchSource:0}: Error finding container 8ade60c1a422914833f60bd84bfbdeee855f59258265d6fba54302b24cc70095: Status 404 returned error can't find the container with id 8ade60c1a422914833f60bd84bfbdeee855f59258265d6fba54302b24cc70095
Nov 03 11:10:42.881027 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:10:42.881002    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3215f33_c070_49bc_ad4b_141c2b11e67d.slice/crio-33f8f1e7c444d99c6fe3019f5a5b65db6f4d79b087972ef2d748690885da024e WatchSource:0}: Error finding container 33f8f1e7c444d99c6fe3019f5a5b65db6f4d79b087972ef2d748690885da024e: Status 404 returned error can't find the container with id 33f8f1e7c444d99c6fe3019f5a5b65db6f4d79b087972ef2d748690885da024e
Nov 03 11:10:42.883005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.882980    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"]
Nov 03 11:10:42.973699 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:10:42.973668    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813cff89_38c5_46db_a92a_187b541220ee.slice/crio-1bb499ee9a8c2550d1e44e0028e8c84cd52b2ee6e9547743879cc868424efce0 WatchSource:0}: Error finding container 1bb499ee9a8c2550d1e44e0028e8c84cd52b2ee6e9547743879cc868424efce0: Status 404 returned error can't find the container with id 1bb499ee9a8c2550d1e44e0028e8c84cd52b2ee6e9547743879cc868424efce0
Nov 03 11:10:42.975059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:42.975033    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"]
Nov 03 11:10:43.237144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.237098    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:43.254114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.254081    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd" event={"ID":"b3215f33-c070-49bc-ad4b-141c2b11e67d","Type":"ContainerStarted","Data":"dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2"}
Nov 03 11:10:43.254279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.254154    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd" event={"ID":"b3215f33-c070-49bc-ad4b-141c2b11e67d","Type":"ContainerStarted","Data":"33f8f1e7c444d99c6fe3019f5a5b65db6f4d79b087972ef2d748690885da024e"}
Nov 03 11:10:43.254841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.254820    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:43.256763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.256738    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj" event={"ID":"9d76f4dd-9618-4ead-8cd2-b999526c1454","Type":"ContainerStarted","Data":"dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b"}
Nov 03 11:10:43.256893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.256785    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj" event={"ID":"9d76f4dd-9618-4ead-8cd2-b999526c1454","Type":"ContainerStarted","Data":"8ade60c1a422914833f60bd84bfbdeee855f59258265d6fba54302b24cc70095"}
Nov 03 11:10:43.256893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.256827    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:43.258515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.258494    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz" event={"ID":"813cff89-38c5-46db-a92a-187b541220ee","Type":"ContainerStarted","Data":"64003934817fa6f978d8a0d68e9e235950f30af5fc215f2fde6a2f0c48f9b434"}
Nov 03 11:10:43.258624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.258538    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz" event={"ID":"813cff89-38c5-46db-a92a-187b541220ee","Type":"ContainerStarted","Data":"1bb499ee9a8c2550d1e44e0028e8c84cd52b2ee6e9547743879cc868424efce0"}
Nov 03 11:10:43.258706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.258690    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:43.261041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.261019    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj" event={"ID":"05f96069-99fe-4409-9415-d674a1da3ed1","Type":"ContainerStarted","Data":"36eb47431727edcd43c11611ceeb86fbb0a6383d1da781d93f48d55d1203f6a9"}
Nov 03 11:10:43.261162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.261135    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:43.263018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.262985    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9" event={"ID":"310d71bb-1549-407b-8ce4-0b0383ca4429","Type":"ContainerDied","Data":"c7539211cd18058704b8d79bc82bb0ce28496776631900ab45dfcd4134cc4a95"}
Nov 03 11:10:43.263129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.262996    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-449q9"
Nov 03 11:10:43.263129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.263037    5817 scope.go:122] "RemoveContainer" containerID="da2e511a02d1abf3ac855517d978c9c09541b64a190f42b745270b807fc3f525"
Nov 03 11:10:43.296489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.296437    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd" podStartSLOduration=2.296425333 podStartE2EDuration="2.296425333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:43.289718664 +0000 UTC m=+2473.495300716" watchObservedRunningTime="2026-11-03 11:10:43.296425333 +0000 UTC m=+2473.502007382"
Nov 03 11:10:43.319268 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.319228    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj" podStartSLOduration=2.319217156 podStartE2EDuration="2.319217156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:43.316089387 +0000 UTC m=+2473.521671433" watchObservedRunningTime="2026-11-03 11:10:43.319217156 +0000 UTC m=+2473.524799206"
Nov 03 11:10:43.338641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.338598    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz" podStartSLOduration=1.338586455 podStartE2EDuration="1.338586455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:43.333082841 +0000 UTC m=+2473.538664893" watchObservedRunningTime="2026-11-03 11:10:43.338586455 +0000 UTC m=+2473.544168497"
Nov 03 11:10:43.354653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.354625    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-449q9"]
Nov 03 11:10:43.363563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.363537    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-449q9"]
Nov 03 11:10:43.371467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.371430    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj" podStartSLOduration=4.371419781 podStartE2EDuration="4.371419781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:43.368799431 +0000 UTC m=+2473.574381483" watchObservedRunningTime="2026-11-03 11:10:43.371419781 +0000 UTC m=+2473.577001832"
Nov 03 11:10:43.476599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.472679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:43.915599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.907101    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:43.991958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:43.991920    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310d71bb-1549-407b-8ce4-0b0383ca4429" path="/var/lib/kubelet/pods/310d71bb-1549-407b-8ce4-0b0383ca4429/volumes"
Nov 03 11:10:44.606032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:44.605985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:45.186756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:45.184203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:45.864596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:45.863724    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:46.573597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:46.573359    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:47.155516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:47.154205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:47.859436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:47.859329    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:48.514025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:48.513990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:48.653287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:48.653256    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:10:49.170797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:49.170761    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:49.855771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:49.855737    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:50.582151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:50.582118    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:51.227531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:51.227495    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:51.903335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:51.903301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:52.418502 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:10:52.416209    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:10:52.563598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:52.563443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:53.241623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:53.237668    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:53.888604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:53.878665    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:54.034683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:54.034647    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:10:54.276539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:54.276505    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:10:54.276760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:54.276586    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:10:54.276760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:54.276632    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:10:54.415282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:54.415240    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"]
Nov 03 11:10:54.774452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:54.774415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:55.520603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:55.519765    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:10:56.526637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:56.525657    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 11:10:56.712063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:56.712021    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.713263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:56.713230    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 11:10:56.735598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:56.735534    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.787561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:56.787442    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.828439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:56.826661    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.964336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:56.963478    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.966036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:56.965229    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:57.195651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:57.183794    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:57.662855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:57.656385    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:58.066604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:58.064619    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"]
Nov 03 11:10:58.076811 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:58.072021    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5a2a639-9lk7g"]
Nov 03 11:10:59.147597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:59.144364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:10:59.440649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:59.435548    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj" podUID="05f96069-99fe-4409-9415-d674a1da3ed1" containerName="payload-processing" containerID="cri-o://36eb47431727edcd43c11611ceeb86fbb0a6383d1da781d93f48d55d1203f6a9" gracePeriod=25
Nov 03 11:10:59.767668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:59.761993    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:10:59.767668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:59.762080    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:59.995694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:10:59.995655    5817 generic.go:356] "Generic (PLEG): container finished" podID="05f96069-99fe-4409-9415-d674a1da3ed1" containerID="36eb47431727edcd43c11611ceeb86fbb0a6383d1da781d93f48d55d1203f6a9" exitCode=0
Nov 03 11:11:00.097519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.082638    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078bb7a8-927c-419e-8b08-f73b4da83154" path="/var/lib/kubelet/pods/078bb7a8-927c-419e-8b08-f73b4da83154/volumes"
Nov 03 11:11:00.097519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.083208    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj" event={"ID":"05f96069-99fe-4409-9415-d674a1da3ed1","Type":"ContainerDied","Data":"36eb47431727edcd43c11611ceeb86fbb0a6383d1da781d93f48d55d1203f6a9"}
Nov 03 11:11:00.427945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.427909    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:11:00.504308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.504255    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-routing-overlay-volume\") pod \"05f96069-99fe-4409-9415-d674a1da3ed1\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") "
Nov 03 11:11:00.506103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.504923    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-routing-overlay-volume" pod "05f96069-99fe-4409-9415-d674a1da3ed1" (UID: "05f96069-99fe-4409-9415-d674a1da3ed1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:00.506448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.506430    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-provider-credentials\") pod \"05f96069-99fe-4409-9415-d674a1da3ed1\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") "
Nov 03 11:11:00.506684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.506668    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-plugins-config-volume\") pod \"05f96069-99fe-4409-9415-d674a1da3ed1\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") "
Nov 03 11:11:00.507409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.507392    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-kube-api-access-26w65\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-kube-api-access-26w65\") pod \"05f96069-99fe-4409-9415-d674a1da3ed1\" (UID: \"05f96069-99fe-4409-9415-d674a1da3ed1\") "
Nov 03 11:11:00.508944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.508922    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:00.509158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.507238    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-plugins-config-volume" pod "05f96069-99fe-4409-9415-d674a1da3ed1" (UID: "05f96069-99fe-4409-9415-d674a1da3ed1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:00.518984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.518944    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-kube-api-access-26w65" pod "05f96069-99fe-4409-9415-d674a1da3ed1" (UID: "05f96069-99fe-4409-9415-d674a1da3ed1"). InnerVolumeSpecName "kube-api-access-26w65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:00.522531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.522498    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-provider-credentials" pod "05f96069-99fe-4409-9415-d674a1da3ed1" (UID: "05f96069-99fe-4409-9415-d674a1da3ed1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:00.612849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.612756    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26w65\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-kube-api-access-26w65\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:00.612849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.612790    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05f96069-99fe-4409-9415-d674a1da3ed1-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:00.612849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:00.612799    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f96069-99fe-4409-9415-d674a1da3ed1-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:01.009864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:01.009825    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj" event={"ID":"05f96069-99fe-4409-9415-d674a1da3ed1","Type":"ContainerDied","Data":"dfa3eb18a4a6cacf54c3ae71b4a8b538e5803479d2332aaa54c4415529caac16"}
Nov 03 11:11:01.010077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:01.009892    5817 scope.go:122] "RemoveContainer" containerID="36eb47431727edcd43c11611ceeb86fbb0a6383d1da781d93f48d55d1203f6a9"
Nov 03 11:11:01.010077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:01.009841    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"
Nov 03 11:11:01.040091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:01.040056    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"]
Nov 03 11:11:01.045877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:01.045845    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855kxfgj"]
Nov 03 11:11:01.274400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:01.274309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:01.908438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:01.908395    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f96069-99fe-4409-9415-d674a1da3ed1" path="/var/lib/kubelet/pods/05f96069-99fe-4409-9415-d674a1da3ed1/volumes"
Nov 03 11:11:01.976795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:01.976759    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:02.662045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:02.661995    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:02.876409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:02.876374    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:03.434595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:03.434506    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:04.066370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:04.066323    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:04.732594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:04.729281    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:05.414354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:05.414319    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:06.068457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:06.068417    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:06.734616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:06.731594    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:07.544385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:07.544354    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:08.375144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:08.375114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:09.048495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:09.048452    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:09.065706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:09.065653    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"]
Nov 03 11:11:09.093713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:09.091047    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1e36d6-m74nj"]
Nov 03 11:11:09.788599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:09.781200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:09.906440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:09.906407    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056fbbb6-d2a5-4a7b-bb7d-543af21d909c" path="/var/lib/kubelet/pods/056fbbb6-d2a5-4a7b-bb7d-543af21d909c/volumes"
Nov 03 11:11:10.451476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.451439    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:10.791769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.791692    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"]
Nov 03 11:11:10.792698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.792674    5817 cpu_manager.go:414] "containerMap: removing container" podUID="05f96069-99fe-4409-9415-d674a1da3ed1" containerName="payload-processing"
Nov 03 11:11:10.792698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.792697    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f96069-99fe-4409-9415-d674a1da3ed1" containerName="payload-processing"
Nov 03 11:11:10.792874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.792745    5817 cpu_manager.go:414] "containerMap: removing container" podUID="310d71bb-1549-407b-8ce4-0b0383ca4429" containerName="payload-processing"
Nov 03 11:11:10.792874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.792750    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310d71bb-1549-407b-8ce4-0b0383ca4429" containerName="payload-processing"
Nov 03 11:11:10.792947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.792876    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="310d71bb-1549-407b-8ce4-0b0383ca4429" containerName="payload-processing"
Nov 03 11:11:10.792947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.792889    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f96069-99fe-4409-9415-d674a1da3ed1" containerName="payload-processing"
Nov 03 11:11:10.798845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.798817    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.803624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.803560    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:11:10.803782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.803655    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:11:10.813851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.813826    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"]
Nov 03 11:11:10.875502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.875468    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9649f\" (UniqueName: \"kubernetes.io/projected/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-kube-api-access-9649f\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.875694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.875548    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.875694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.875669    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.977257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.977223    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.977452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.977303    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.977689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.977667    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9649f\" (UniqueName: \"kubernetes.io/projected/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-kube-api-access-9649f\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.977944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.977913    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.979928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.979909    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:10.986328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:10.986304    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9649f\" (UniqueName: \"kubernetes.io/projected/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-kube-api-access-9649f\") pod \"maas-api-revoke-sub-12b19af0-7mcsb\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:11.120069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:11.119962    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:11.295362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:11.295176    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:11.410188 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:11:11.410160    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c919e1_fde6_4f1c_a5cf_80fcc9712fa9.slice/crio-a9c37874f78b111a448270997b5fee8f4b3e5da7fd70b70d770811d4267c709e WatchSource:0}: Error finding container a9c37874f78b111a448270997b5fee8f4b3e5da7fd70b70d770811d4267c709e: Status 404 returned error can't find the container with id a9c37874f78b111a448270997b5fee8f4b3e5da7fd70b70d770811d4267c709e
Nov 03 11:11:11.412654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:11.412629    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"]
Nov 03 11:11:11.928245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:11.928214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:12.190673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:12.190631    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb" event={"ID":"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9","Type":"ContainerStarted","Data":"0a6d32544b229637c148875a1edcbd9b24816bc58d000dc581910733244027af"}
Nov 03 11:11:12.190875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:12.190702    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb" event={"ID":"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9","Type":"ContainerStarted","Data":"a9c37874f78b111a448270997b5fee8f4b3e5da7fd70b70d770811d4267c709e"}
Nov 03 11:11:12.212729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:12.212677    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb" podStartSLOduration=2.212664186 podStartE2EDuration="2.212664186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:11:12.209928474 +0000 UTC m=+2502.415510518" watchObservedRunningTime="2026-11-03 11:11:12.212664186 +0000 UTC m=+2502.418246237"
Nov 03 11:11:12.560252 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:12.560122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:13.208059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:13.208016    5817 generic.go:356] "Generic (PLEG): container finished" podID="e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9" containerID="0a6d32544b229637c148875a1edcbd9b24816bc58d000dc581910733244027af" exitCode=0
Nov 03 11:11:13.208507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:13.208130    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb" event={"ID":"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9","Type":"ContainerDied","Data":"0a6d32544b229637c148875a1edcbd9b24816bc58d000dc581910733244027af"}
Nov 03 11:11:13.273598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:13.270382    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:13.870965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:13.870927    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:14.392870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.392836    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:14.488896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.488807    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-openshift-ca\") pod \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") "
Nov 03 11:11:14.489073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.488993    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-kube-api-access-9649f\" (UniqueName: \"kubernetes.io/projected/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-kube-api-access-9649f\") pod \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") "
Nov 03 11:11:14.489173 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.489101    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-xks-ca\") pod \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\" (UID: \"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9\") "
Nov 03 11:11:14.489286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.489256    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-openshift-ca" pod "e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9" (UID: "e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:14.490852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.490176    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:14.499610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.493718    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-xks-ca" pod "e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9" (UID: "e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:11:14.499610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.493842    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-kube-api-access-9649f" pod "e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9" (UID: "e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9"). InnerVolumeSpecName "kube-api-access-9649f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:14.615597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.614678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:14.637596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.616536    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9649f\" (UniqueName: \"kubernetes.io/projected/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-kube-api-access-9649f\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:14.642592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:14.641641    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:15.232874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:15.232840    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"
Nov 03 11:11:15.233042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:15.232873    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb" event={"ID":"e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9","Type":"ContainerDied","Data":"a9c37874f78b111a448270997b5fee8f4b3e5da7fd70b70d770811d4267c709e"}
Nov 03 11:11:15.233042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:15.232926    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9c37874f78b111a448270997b5fee8f4b3e5da7fd70b70d770811d4267c709e"
Nov 03 11:11:15.431452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:15.431409    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:16.186928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:16.186801    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:16.896595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:16.886491    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:17.581302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:17.581267    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:18.156283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:18.156251    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:18.313485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:18.313451    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:18.964428 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:18.964373    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:19.594599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:19.593838    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:20.304621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:20.291238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:21.197642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:21.197608    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:21.686739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:21.686695    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"]
Nov 03 11:11:21.765550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:21.765518    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:21.907594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:21.903280    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:22.183602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:22.167872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:22.399646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:22.390655    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:22.522174 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:11:22.522027    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:11:23.220595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:23.209508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:23.886606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:23.883607    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:24.560738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:24.560702    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:25.208162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:25.208127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:25.878280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:25.878243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:26.485608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:26.485517    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:26.687219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:26.687171    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g" podUID="cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" containerName="payload-processing" containerID="cri-o://70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2" gracePeriod=25
Nov 03 11:11:26.991529 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:26.991499    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:11:27.166914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.166402    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-provider-credentials\") pod \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") "
Nov 03 11:11:27.166914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.166507    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-plugins-config-volume\") pod \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") "
Nov 03 11:11:27.166914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.166759    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-kube-api-access-zb5wf\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-kube-api-access-zb5wf\") pod \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") "
Nov 03 11:11:27.166914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.166845    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-routing-overlay-volume\") pod \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\" (UID: \"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5\") "
Nov 03 11:11:27.168049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.168020    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-routing-overlay-volume" pod "cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" (UID: "cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:27.173817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.172600    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-plugins-config-volume" pod "cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" (UID: "cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:27.173817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.172742    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-provider-credentials" pod "cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" (UID: "cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:27.179070 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.175654    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:27.194141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.194108    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-kube-api-access-zb5wf" pod "cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" (UID: "cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5"). InnerVolumeSpecName "kube-api-access-zb5wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:27.269228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.269190    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:27.269228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.269227    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:27.269484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.269246    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb5wf\" (UniqueName: \"kubernetes.io/projected/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-kube-api-access-zb5wf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:27.269484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.269262    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:27.566818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.566786    5817 generic.go:356] "Generic (PLEG): container finished" podID="cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" containerID="70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2" exitCode=0
Nov 03 11:11:27.567270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.566836    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"
Nov 03 11:11:27.567270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.566864    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g" event={"ID":"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5","Type":"ContainerDied","Data":"70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2"}
Nov 03 11:11:27.567270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.566919    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g" event={"ID":"cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5","Type":"ContainerDied","Data":"c28bd9e76a91fd868ee18d5c10887fcb27dac8b9f13c87c940f84c81882f1cec"}
Nov 03 11:11:27.567270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.566944    5817 scope.go:122] "RemoveContainer" containerID="70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2"
Nov 03 11:11:27.579527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.579504    5817 scope.go:122] "RemoveContainer" containerID="70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2"
Nov 03 11:11:27.579843 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:11:27.579820    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2\": container with ID starting with 70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2 not found: ID does not exist" containerID="70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2"
Nov 03 11:11:27.579942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.579850    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2"} err="failed to get container status \"70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2\": rpc error: code = NotFound desc = could not find container \"70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2\": container with ID starting with 70d22d3c4457bcbfeea3d454be1895046b089e509455a74977bc6c613ab2b5d2 not found: ID does not exist"
Nov 03 11:11:27.597062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.597032    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"]
Nov 03 11:11:27.605131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.605103    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g"]
Nov 03 11:11:27.834349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.834252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:27.905402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.905367    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" path="/var/lib/kubelet/pods/cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5/volumes"
Nov 03 11:11:27.949798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:27.949757    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-d6f4rjz5g" podUID="cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.4:50052\" within 1s: context deadline exceeded"
Nov 03 11:11:27.949999 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:11:27.949857    5817 logging.go:55] [core] [Channel #8603 SubChannel #8604]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.4:50052", ServerName: "10.133.1.4:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.4:50052: operation was canceled"
Nov 03 11:11:28.768322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:28.768285    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:28.941444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:28.941410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:29.071643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:29.065636    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"]
Nov 03 11:11:29.119633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:29.116117    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"]
Nov 03 11:11:29.153606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:29.153176    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5c93f0-m5s4p"]
Nov 03 11:11:29.186599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:29.185711    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0888a862-rrm7c"]
Nov 03 11:11:29.696328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:29.696294    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:29.907520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:29.907478    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29dd88e4-d5e5-4e33-8d40-0edadb5d114e" path="/var/lib/kubelet/pods/29dd88e4-d5e5-4e33-8d40-0edadb5d114e/volumes"
Nov 03 11:11:29.908057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:29.908040    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90589240-73ab-41cb-a847-bd57a47a0433" path="/var/lib/kubelet/pods/90589240-73ab-41cb-a847-bd57a47a0433/volumes"
Nov 03 11:11:30.830559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:30.830522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:32.195613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:32.191061    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:32.986931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:32.986892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:33.741118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:33.741082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:34.032444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:34.032362    5817 scope.go:122] "RemoveContainer" containerID="d5e3efe8c1e91cfbefbd614c85201bbbca31a460f0ce9e55c08c7eb602d02e42"
Nov 03 11:11:34.363650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:34.359479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:34.998141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:34.998102    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:35.220592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:35.220535    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:35.448452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:35.448334    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:35.628243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:35.628202    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:35.879192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:35.879087    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:36.021034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:36.020226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:36.178664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:36.177757    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:36.326964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:36.326935    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:36.695424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:36.687311    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:37.378430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:37.366073    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:38.157456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:38.157419    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:38.850593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:38.849672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:39.492648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:39.488702    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:40.242393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:40.242352    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:40.783299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:40.783263    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:41.025186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:41.025151    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"]
Nov 03 11:11:41.030446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:41.030420    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d9fb22-srz5g"]
Nov 03 11:11:41.386602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:41.386529    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:41.907984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:41.907937    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10fe2b3-0973-4845-ad38-ac0d08658652" path="/var/lib/kubelet/pods/d10fe2b3-0973-4845-ad38-ac0d08658652/volumes"
Nov 03 11:11:42.119787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:42.119747    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:42.280530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:42.280492    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:42.426515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:42.426469    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:43.190597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.190205    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"]
Nov 03 11:11:43.220597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.191478    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" containerName="payload-processing"
Nov 03 11:11:43.220597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.191504    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" containerName="payload-processing"
Nov 03 11:11:43.220597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.191620    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9" containerName="revoke-keys"
Nov 03 11:11:43.220597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.191629    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9" containerName="revoke-keys"
Nov 03 11:11:43.220597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.191853    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9" containerName="revoke-keys"
Nov 03 11:11:43.220597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.191876    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee7a78e-1676-4dcd-b4a1-b515fd7f3bd5" containerName="payload-processing"
Nov 03 11:11:43.220597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.216246    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:43.395682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.387908    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.410604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.407503    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:11:43.438999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.435156    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:11:43.440544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.440472    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"]
Nov 03 11:11:43.489460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.489426    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.490029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.490004    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvdf\" (UniqueName: \"kubernetes.io/projected/ce303c9a-dd89-4538-bebd-cda0494cf78e-kube-api-access-stvdf\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.490249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.490233    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.494635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.494033    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:43.544317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.534687    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"]
Nov 03 11:11:43.586059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.586027    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"]
Nov 03 11:11:43.586262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.586146    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.590839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.590815    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b76c62\""
Nov 03 11:11:43.593293 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.593274    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"openshift-service-ca.crt\""
Nov 03 11:11:43.593480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.593467    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"routing-overlay\""
Nov 03 11:11:43.593632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.593619    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"agc-cred-ea38c5efe071\""
Nov 03 11:11:43.593816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.593804    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"kube-root-ca.crt\""
Nov 03 11:11:43.596269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.596232    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvdf\" (UniqueName: \"kubernetes.io/projected/ce303c9a-dd89-4538-bebd-cda0494cf78e-kube-api-access-stvdf\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.596761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.596742    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.596997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.596983    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.598288 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.598267    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.644809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.635280    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.682734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.682701    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvdf\" (UniqueName: \"kubernetes.io/projected/ce303c9a-dd89-4538-bebd-cda0494cf78e-kube-api-access-stvdf\") pod \"maas-api-revoke-sub-6c1c3efd-c8lvb\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.721167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.721087    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:43.736595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.728849    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.736595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.729031    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.736595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.729122    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdckc\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-kube-api-access-kdckc\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.736595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.729157    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.803591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.798902    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:43.860601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.860368    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.860601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.860561    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.860868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.860679    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdckc\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-kube-api-access-kdckc\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.860868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.860713    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.874606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.861538    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.874606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.866499    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.942595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.907091    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:43.956641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:43.954284    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdckc\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-kube-api-access-kdckc\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:44.068662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:44.068545    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"]
Nov 03 11:11:44.073222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:44.073192    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1540dc2-4dkxt"]
Nov 03 11:11:44.156626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:44.156583    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:44.287597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:44.279410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:44.331250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:44.330781    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"]
Nov 03 11:11:44.454060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:44.454032    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"]
Nov 03 11:11:44.490158 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:11:44.490088    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b083a5_e5e2_45e9_a598_85db51df4429.slice/crio-c4690b78211a701a3d7a55ab6d775a2bd8efb5efdcae0a72ccd2eb9b5d81d60c WatchSource:0}: Error finding container c4690b78211a701a3d7a55ab6d775a2bd8efb5efdcae0a72ccd2eb9b5d81d60c: Status 404 returned error can't find the container with id c4690b78211a701a3d7a55ab6d775a2bd8efb5efdcae0a72ccd2eb9b5d81d60c
Nov 03 11:11:44.961370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:44.961339    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:45.087635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.087526    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb" event={"ID":"ce303c9a-dd89-4538-bebd-cda0494cf78e","Type":"ContainerStarted","Data":"2dbd965b4b04c2fd2493eda6abc08aed61edad6fe1ddcc4c5b89b7d1a0aaab5a"}
Nov 03 11:11:45.087635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.087612    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb" event={"ID":"ce303c9a-dd89-4538-bebd-cda0494cf78e","Type":"ContainerStarted","Data":"5e739202d6fe89349217e62ae447c78518c4153ffc7991dd5b47b7319062bf49"}
Nov 03 11:11:45.090466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.090016    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp" event={"ID":"37b083a5-e5e2-45e9-a598-85db51df4429","Type":"ContainerStarted","Data":"700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822"}
Nov 03 11:11:45.090466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.090068    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp" event={"ID":"37b083a5-e5e2-45e9-a598-85db51df4429","Type":"ContainerStarted","Data":"c4690b78211a701a3d7a55ab6d775a2bd8efb5efdcae0a72ccd2eb9b5d81d60c"}
Nov 03 11:11:45.090466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.090104    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:45.109921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.109878    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb" podStartSLOduration=2.109865916 podStartE2EDuration="2.109865916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:11:45.103910081 +0000 UTC m=+2535.309492134" watchObservedRunningTime="2026-11-03 11:11:45.109865916 +0000 UTC m=+2535.315448003"
Nov 03 11:11:45.121675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.121633    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp" podStartSLOduration=2.121621039 podStartE2EDuration="2.121621039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:11:45.118660229 +0000 UTC m=+2535.324242281" watchObservedRunningTime="2026-11-03 11:11:45.121621039 +0000 UTC m=+2535.327203089"
Nov 03 11:11:45.743646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.743301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:45.906142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:45.906108    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e695ab-5123-4d8d-a35f-3525b0d114d2" path="/var/lib/kubelet/pods/06e695ab-5123-4d8d-a35f-3525b0d114d2/volumes"
Nov 03 11:11:46.104688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:46.104601    5817 generic.go:356] "Generic (PLEG): container finished" podID="ce303c9a-dd89-4538-bebd-cda0494cf78e" containerID="2dbd965b4b04c2fd2493eda6abc08aed61edad6fe1ddcc4c5b89b7d1a0aaab5a" exitCode=0
Nov 03 11:11:46.104688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:46.104678    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb" event={"ID":"ce303c9a-dd89-4538-bebd-cda0494cf78e","Type":"ContainerDied","Data":"2dbd965b4b04c2fd2493eda6abc08aed61edad6fe1ddcc4c5b89b7d1a0aaab5a"}
Nov 03 11:11:46.403656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:46.396874    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:47.044601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.044230    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:47.320628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.315072    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:47.391966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.391932    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-xks-ca\") pod \"ce303c9a-dd89-4538-bebd-cda0494cf78e\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") "
Nov 03 11:11:47.392192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.392174    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce303c9a-dd89-4538-bebd-cda0494cf78e-kube-api-access-stvdf\" (UniqueName: \"kubernetes.io/projected/ce303c9a-dd89-4538-bebd-cda0494cf78e-kube-api-access-stvdf\") pod \"ce303c9a-dd89-4538-bebd-cda0494cf78e\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") "
Nov 03 11:11:47.394391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.394366    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-xks-ca" pod "ce303c9a-dd89-4538-bebd-cda0494cf78e" (UID: "ce303c9a-dd89-4538-bebd-cda0494cf78e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:11:47.394930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.394906    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce303c9a-dd89-4538-bebd-cda0494cf78e-kube-api-access-stvdf" pod "ce303c9a-dd89-4538-bebd-cda0494cf78e" (UID: "ce303c9a-dd89-4538-bebd-cda0494cf78e"). InnerVolumeSpecName "kube-api-access-stvdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:47.498480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.497278    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-openshift-ca\") pod \"ce303c9a-dd89-4538-bebd-cda0494cf78e\" (UID: \"ce303c9a-dd89-4538-bebd-cda0494cf78e\") "
Nov 03 11:11:47.498480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.498139    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvdf\" (UniqueName: \"kubernetes.io/projected/ce303c9a-dd89-4538-bebd-cda0494cf78e-kube-api-access-stvdf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:47.498480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.498164    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:47.499035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.498993    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-openshift-ca" pod "ce303c9a-dd89-4538-bebd-cda0494cf78e" (UID: "ce303c9a-dd89-4538-bebd-cda0494cf78e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:47.599412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.599332    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce303c9a-dd89-4538-bebd-cda0494cf78e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:47.880931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:47.880827    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:48.132420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:48.132339    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"
Nov 03 11:11:48.132420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:48.132378    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb" event={"ID":"ce303c9a-dd89-4538-bebd-cda0494cf78e","Type":"ContainerDied","Data":"5e739202d6fe89349217e62ae447c78518c4153ffc7991dd5b47b7319062bf49"}
Nov 03 11:11:48.132991 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:48.132425    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e739202d6fe89349217e62ae447c78518c4153ffc7991dd5b47b7319062bf49"
Nov 03 11:11:48.555256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:48.554989    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:49.202613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:49.197203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:49.981382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:49.981352    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:50.414441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:50.414347    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:11:50.701598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:50.698467    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:51.778597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:51.756200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:52.418806 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:11:52.418765    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:11:52.750304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:52.750270    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:53.686607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:53.684096    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:54.029300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:54.029197    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"]
Nov 03 11:11:54.035829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:54.035798    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cea063-8nrdh"]
Nov 03 11:11:54.209700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:54.209666    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"]
Nov 03 11:11:55.284280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:55.284237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:55.418343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:55.418303    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:55.728606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:55.723904    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:55.885333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:55.885299    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:55.924960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:55.924926    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8713a290-7ad4-464e-9ff5-be42069ce4b6" path="/var/lib/kubelet/pods/8713a290-7ad4-464e-9ff5-be42069ce4b6/volumes"
Nov 03 11:11:56.026088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:56.025993    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:56.108457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:56.108433    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:11:56.171197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:56.171157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:56.446782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:56.446746    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:56.600717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:56.600661    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:56.996538 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:56.996502    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:57.028409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.028381    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"]
Nov 03 11:11:57.038279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.038253    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76f7ab9-ddj9v"]
Nov 03 11:11:57.143056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.143014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:57.628115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.628074    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:57.635329 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.635302    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"]
Nov 03 11:11:57.636900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.636859    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ce303c9a-dd89-4538-bebd-cda0494cf78e" containerName="revoke-keys"
Nov 03 11:11:57.637438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.637419    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce303c9a-dd89-4538-bebd-cda0494cf78e" containerName="revoke-keys"
Nov 03 11:11:57.638079 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.638060    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce303c9a-dd89-4538-bebd-cda0494cf78e" containerName="revoke-keys"
Nov 03 11:11:57.645915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.645892    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"]
Nov 03 11:11:57.646028 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.646000    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.650238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.650204    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"openshift-service-ca.crt\""
Nov 03 11:11:57.650868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.650464    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"kube-root-ca.crt\""
Nov 03 11:11:57.652987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.651870    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"agc-cred-8b36c5cf596c\""
Nov 03 11:11:57.652987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.651999    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"payload-processing-external-model-e2e--885106fd-dockercfg-t8tsb\""
Nov 03 11:11:57.652987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.652059    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"routing-overlay\""
Nov 03 11:11:57.652987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.652096    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-56687c\""
Nov 03 11:11:57.855596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.855264    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.855596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.855589    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrxx\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-kube-api-access-kkrxx\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.855893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.855652    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.855893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.855683    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.977670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.958722    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.978201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.978175    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.978500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.978477    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.979013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.978990    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrxx\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-kube-api-access-kkrxx\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.980234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.980209    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:57.981188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:57.981165    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:58.035672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.023107    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:58.036964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.036899    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrxx\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-kube-api-access-kkrxx\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:58.118655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.117004    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:58.120858 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.118920    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f505ecce-7e6c-4ab6-8165-1ee72f8dc40e" path="/var/lib/kubelet/pods/f505ecce-7e6c-4ab6-8165-1ee72f8dc40e/volumes"
Nov 03 11:11:58.120858 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.119464    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"]
Nov 03 11:11:58.127832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.127811    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.150200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.150120    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"payload-processing-external-model-e2e--c9e9a1ee-dockercfg-nd2vw\""
Nov 03 11:11:58.155470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.155372    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"kube-root-ca.crt\""
Nov 03 11:11:58.155651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.155528    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"openshift-service-ca.crt\""
Nov 03 11:11:58.155728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.155670    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"routing-overlay\""
Nov 03 11:11:58.155805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.155784    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"payload-processing-external-model-plugins-e2e-extmod-a-185908\""
Nov 03 11:11:58.417845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.287743    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"agc-cred-5eb63f008755\""
Nov 03 11:11:58.544987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.543132    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:58.544987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.543833    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"]
Nov 03 11:11:58.545214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.545124    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.545893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.545450    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.545893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.545587    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.545893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.545641    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpc42\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-kube-api-access-bpc42\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.648240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.648211    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.648240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.648332    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.648240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.648375    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpc42\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-kube-api-access-bpc42\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.648921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.648479    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.649489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.649449    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.653146 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.653116    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.654631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.654605    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.660678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.660620    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpc42\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-kube-api-access-bpc42\") pod \"payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.845596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.845541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:58.893637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.888723    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:58.918531 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:11:58.918495    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ed7c9c_5ca8_4974_9c3d_fff3c48fc090.slice/crio-92bdc30f4051bf273abf3059639cb2b207a17675023851d09a631b9ed0943c9e WatchSource:0}: Error finding container 92bdc30f4051bf273abf3059639cb2b207a17675023851d09a631b9ed0943c9e: Status 404 returned error can't find the container with id 92bdc30f4051bf273abf3059639cb2b207a17675023851d09a631b9ed0943c9e
Nov 03 11:11:58.919268 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:58.919236    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"]
Nov 03 11:11:59.202031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.202003    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"]
Nov 03 11:11:59.216762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.210687    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2" podUID="10493144-b203-457e-8e5e-b8419e8fe53b" containerName="payload-processing" containerID="cri-o://544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737" gracePeriod=25
Nov 03 11:11:59.538597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.536847    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:59.611453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.611419    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:11:59.611782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.611759    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" event={"ID":"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090","Type":"ContainerStarted","Data":"155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec"}
Nov 03 11:11:59.611870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.611805    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" event={"ID":"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090","Type":"ContainerStarted","Data":"92bdc30f4051bf273abf3059639cb2b207a17675023851d09a631b9ed0943c9e"}
Nov 03 11:11:59.612030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.612012    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:11:59.613908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.613880    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" event={"ID":"cad4340e-0da6-4664-825f-cfbba8dc0017","Type":"ContainerStarted","Data":"5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080"}
Nov 03 11:11:59.614059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.613923    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" event={"ID":"cad4340e-0da6-4664-825f-cfbba8dc0017","Type":"ContainerStarted","Data":"c85cbe3fce88c9b7beb997708b6ec528f85434ab655e0de02ae17b9af59a52d9"}
Nov 03 11:11:59.614059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.613959    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:11:59.615531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.615504    5817 generic.go:356] "Generic (PLEG): container finished" podID="10493144-b203-457e-8e5e-b8419e8fe53b" containerID="544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737" exitCode=0
Nov 03 11:11:59.615679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.615534    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2" event={"ID":"10493144-b203-457e-8e5e-b8419e8fe53b","Type":"ContainerDied","Data":"544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737"}
Nov 03 11:11:59.615679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.615593    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2" event={"ID":"10493144-b203-457e-8e5e-b8419e8fe53b","Type":"ContainerDied","Data":"6c4401424ab499bcf6c20c741835011d2c0be4189c8f401b66593fdacc2229ab"}
Nov 03 11:11:59.615679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.615543    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"
Nov 03 11:11:59.615679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.615622    5817 scope.go:122] "RemoveContainer" containerID="544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737"
Nov 03 11:11:59.628646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.628628    5817 scope.go:122] "RemoveContainer" containerID="544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737"
Nov 03 11:11:59.628944 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:11:59.628926    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737\": container with ID starting with 544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737 not found: ID does not exist" containerID="544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737"
Nov 03 11:11:59.629014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.628952    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737"} err="failed to get container status \"544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737\": rpc error: code = NotFound desc = could not find container \"544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737\": container with ID starting with 544435d01a8d49df9a644fde45014b8a52adac93d42665b85abe3706f1594737 not found: ID does not exist"
Nov 03 11:11:59.652522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.652479    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" podStartSLOduration=2.6524677580000002 podStartE2EDuration="2.652467758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:11:59.648317462 +0000 UTC m=+2549.853899516" watchObservedRunningTime="2026-11-03 11:11:59.652467758 +0000 UTC m=+2549.858049808"
Nov 03 11:11:59.677226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.677149    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" podStartSLOduration=1.677136653 podStartE2EDuration="1.677136653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:11:59.672419936 +0000 UTC m=+2549.878001988" watchObservedRunningTime="2026-11-03 11:11:59.677136653 +0000 UTC m=+2549.882718704"
Nov 03 11:11:59.751698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.751666    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"]
Nov 03 11:11:59.776338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.776303    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-routing-overlay-volume\") pod \"10493144-b203-457e-8e5e-b8419e8fe53b\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") "
Nov 03 11:11:59.776511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.776468    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-provider-credentials\") pod \"10493144-b203-457e-8e5e-b8419e8fe53b\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") "
Nov 03 11:11:59.776511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.776500    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-kube-api-access-6qh86\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-kube-api-access-6qh86\") pod \"10493144-b203-457e-8e5e-b8419e8fe53b\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") "
Nov 03 11:11:59.776690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.776560    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-plugins-config-volume\") pod \"10493144-b203-457e-8e5e-b8419e8fe53b\" (UID: \"10493144-b203-457e-8e5e-b8419e8fe53b\") "
Nov 03 11:11:59.776690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.776658    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-routing-overlay-volume" pod "10493144-b203-457e-8e5e-b8419e8fe53b" (UID: "10493144-b203-457e-8e5e-b8419e8fe53b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:59.777054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.777029    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-plugins-config-volume" pod "10493144-b203-457e-8e5e-b8419e8fe53b" (UID: "10493144-b203-457e-8e5e-b8419e8fe53b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:59.777618 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:11:59.777361    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56687c/agc-cred-8b36c5cf596c: secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:11:59.777618 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:11:59.777387    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w: secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:11:59.777618 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:11:59.777453    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials podName:12ed7c9c-5ca8-4974-9c3d-fff3c48fc090 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:00.277432785 +0000 UTC m=+2550.483014831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090") : secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:11:59.777618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.777470    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:59.777618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.777481    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10493144-b203-457e-8e5e-b8419e8fe53b-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:59.778815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.778796    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-provider-credentials" pod "10493144-b203-457e-8e5e-b8419e8fe53b" (UID: "10493144-b203-457e-8e5e-b8419e8fe53b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:59.778874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.778853    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-kube-api-access-6qh86" pod "10493144-b203-457e-8e5e-b8419e8fe53b" (UID: "10493144-b203-457e-8e5e-b8419e8fe53b"). InnerVolumeSpecName "kube-api-access-6qh86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:59.841267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.841235    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:11:59.879134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.879105    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:59.879134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.879128    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qh86\" (UniqueName: \"kubernetes.io/projected/10493144-b203-457e-8e5e-b8419e8fe53b-kube-api-access-6qh86\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:59.970607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:11:59.964362    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"]
Nov 03 11:12:00.031009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:00.027923    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:00.198596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:00.193614    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68fb489997-rw4h2"]
Nov 03 11:12:00.198596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:00.193688    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"]
Nov 03 11:12:00.260621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:00.259173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:00.326239 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.326150    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56687c/agc-cred-8b36c5cf596c: secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:00.326239 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.326181    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w: secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:00.326489 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.326255    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials podName:12ed7c9c-5ca8-4974-9c3d-fff3c48fc090 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:01.32623371 +0000 UTC m=+2551.531815741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090") : secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:00.532596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:00.531403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:00.747599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:00.747060    5817 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" secret="" err="secret \"payload-processing-external-model-e2e--c9e9a1ee-dockercfg-nd2vw\" not found"
Nov 03 11:12:00.747599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:00.747303    5817 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" secret="" err="secret \"payload-processing-external-model-e2e--885106fd-dockercfg-t8tsb\" not found"
Nov 03 11:12:00.886344 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.886307    5817 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-plugins-e2e-extmod-b-56687c: configmap "payload-processing-external-model-plugins-e2e-extmod-b-56687c" not found
Nov 03 11:12:00.886619 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.886404    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume podName:12ed7c9c-5ca8-4974-9c3d-fff3c48fc090 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:01.386381117 +0000 UTC m=+2551.591963164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-56687c" not found
Nov 03 11:12:00.886619 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.886425    5817 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-plugins-e2e-extmod-a-185908: configmap "payload-processing-external-model-plugins-e2e-extmod-a-185908" not found
Nov 03 11:12:00.886619 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.886450    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-185908/agc-cred-5eb63f008755: secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:00.886619 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.886469    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr: secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:00.886619 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.886490    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume podName:cad4340e-0da6-4664-825f-cfbba8dc0017 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:01.386473361 +0000 UTC m=+2551.592055406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-185908" not found
Nov 03 11:12:00.886619 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:00.886516    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials podName:cad4340e-0da6-4664-825f-cfbba8dc0017 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:01.386501746 +0000 UTC m=+2551.592083792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017") : secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:01.137717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:01.137034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:01.397170 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397087    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56687c/agc-cred-8b36c5cf596c: secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:01.397170 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397110    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w: secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:01.397170 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397142    5817 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-plugins-e2e-extmod-b-56687c: configmap "payload-processing-external-model-plugins-e2e-extmod-b-56687c" not found
Nov 03 11:12:01.397170 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397160    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials podName:12ed7c9c-5ca8-4974-9c3d-fff3c48fc090 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:03.397145382 +0000 UTC m=+2553.602727411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090") : secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:01.397526 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397203    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume podName:12ed7c9c-5ca8-4974-9c3d-fff3c48fc090 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:02.397185632 +0000 UTC m=+2552.602767667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-56687c" not found
Nov 03 11:12:01.397526 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397225    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-185908/agc-cred-5eb63f008755: secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:01.397526 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397245    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr: secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:01.397526 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397295    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials podName:cad4340e-0da6-4664-825f-cfbba8dc0017 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:02.39728032 +0000 UTC m=+2552.602862369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017") : secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:01.397526 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397456    5817 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-plugins-e2e-extmod-a-185908: configmap "payload-processing-external-model-plugins-e2e-extmod-a-185908" not found
Nov 03 11:12:01.397526 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:01.397512    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume podName:cad4340e-0da6-4664-825f-cfbba8dc0017 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:02.397498911 +0000 UTC m=+2552.603080960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-185908" not found
Nov 03 11:12:01.836667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:01.836614    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:01.906607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:01.906539    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10493144-b203-457e-8e5e-b8419e8fe53b" path="/var/lib/kubelet/pods/10493144-b203-457e-8e5e-b8419e8fe53b/volumes"
Nov 03 11:12:02.445565 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:02.445523    5817 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-plugins-e2e-extmod-b-56687c: configmap "payload-processing-external-model-plugins-e2e-extmod-b-56687c" not found
Nov 03 11:12:02.445816 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:02.445632    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume podName:12ed7c9c-5ca8-4974-9c3d-fff3c48fc090 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:04.445609827 +0000 UTC m=+2554.651191874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-56687c" not found
Nov 03 11:12:02.445816 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:02.445658    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-185908/agc-cred-5eb63f008755: secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:02.445816 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:02.445680    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr: secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:02.445816 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:02.445723    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials podName:cad4340e-0da6-4664-825f-cfbba8dc0017 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:04.445711196 +0000 UTC m=+2554.651293244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017") : secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:02.445816 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:02.445655    5817 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-plugins-e2e-extmod-a-185908: configmap "payload-processing-external-model-plugins-e2e-extmod-a-185908" not found
Nov 03 11:12:02.445816 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:02.445753    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume podName:cad4340e-0da6-4664-825f-cfbba8dc0017 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:04.445747308 +0000 UTC m=+2554.651329336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-185908" not found
Nov 03 11:12:02.503990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:02.503942    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:02.919020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:02.918917    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:03.234324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:03.234292    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:03.467406 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:03.467363    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56687c/agc-cred-8b36c5cf596c: secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:03.467406 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:03.467400    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w: secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:03.467646 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:03.467455    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials podName:12ed7c9c-5ca8-4974-9c3d-fff3c48fc090 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:07.467439953 +0000 UTC m=+2557.673021999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090") : secret "agc-cred-8b36c5cf596c" not found
Nov 03 11:12:03.972639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:03.970156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:04.491017 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:04.490979    5817 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-plugins-e2e-extmod-a-185908: configmap "payload-processing-external-model-plugins-e2e-extmod-a-185908" not found
Nov 03 11:12:04.491184 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:04.491070    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume podName:cad4340e-0da6-4664-825f-cfbba8dc0017 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:08.491045592 +0000 UTC m=+2558.696627636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-185908" not found
Nov 03 11:12:04.491381 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:04.491364    5817 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-plugins-e2e-extmod-b-56687c: configmap "payload-processing-external-model-plugins-e2e-extmod-b-56687c" not found
Nov 03 11:12:04.491428 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:04.491416    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume podName:12ed7c9c-5ca8-4974-9c3d-fff3c48fc090 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:08.491405218 +0000 UTC m=+2558.696987265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-56687c" not found
Nov 03 11:12:04.491499 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:04.491486    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-185908/agc-cred-5eb63f008755: secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:04.491534 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:04.491501    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr: secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:04.491565 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:04.491551    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials podName:cad4340e-0da6-4664-825f-cfbba8dc0017 nodeName:}" failed. No retries permitted until 2026-11-03 11:12:08.491530691 +0000 UTC m=+2558.697112736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017") : secret "agc-cred-5eb63f008755" not found
Nov 03 11:12:04.577592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:04.576134    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:05.297886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:05.297830    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:05.986366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:05.986328    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:06.477461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:06.477431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:06.626185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:06.625985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:06.869342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:06.869252    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" podUID="cad4340e-0da6-4664-825f-cfbba8dc0017" containerName="payload-processing" containerID="cri-o://5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080" gracePeriod=25
Nov 03 11:12:06.869342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:06.869269    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" podUID="12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" containerName="payload-processing" containerID="cri-o://155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec" gracePeriod=25
Nov 03 11:12:07.029682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.029641    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:07.210845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.210815    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:12:07.248719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.248690    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-kube-api-access-kkrxx\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-kube-api-access-kkrxx\") pod \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") "
Nov 03 11:12:07.248877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.248856    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials\") pod \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") "
Nov 03 11:12:07.248959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.248908    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-routing-overlay-volume\") pod \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") "
Nov 03 11:12:07.250190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.250162    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-routing-overlay-volume" pod "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:12:07.252609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.252543    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-kube-api-access-kkrxx" pod "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090"). InnerVolumeSpecName "kube-api-access-kkrxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:07.253726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.253703    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials" pod "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:07.264887 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.264865    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:12:07.298368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.298339    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"]
Nov 03 11:12:07.300738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.300713    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cad4340e-0da6-4664-825f-cfbba8dc0017" containerName="payload-processing"
Nov 03 11:12:07.300911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.300888    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad4340e-0da6-4664-825f-cfbba8dc0017" containerName="payload-processing"
Nov 03 11:12:07.301012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.300925    5817 cpu_manager.go:414] "containerMap: removing container" podUID="12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" containerName="payload-processing"
Nov 03 11:12:07.301012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.300934    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" containerName="payload-processing"
Nov 03 11:12:07.301012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.300981    5817 cpu_manager.go:414] "containerMap: removing container" podUID="10493144-b203-457e-8e5e-b8419e8fe53b" containerName="payload-processing"
Nov 03 11:12:07.301012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.300989    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10493144-b203-457e-8e5e-b8419e8fe53b" containerName="payload-processing"
Nov 03 11:12:07.301224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.301211    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="10493144-b203-457e-8e5e-b8419e8fe53b" containerName="payload-processing"
Nov 03 11:12:07.301258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.301240    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad4340e-0da6-4664-825f-cfbba8dc0017" containerName="payload-processing"
Nov 03 11:12:07.301258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.301252    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" containerName="payload-processing"
Nov 03 11:12:07.306647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.306624    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.328435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.328405    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"]
Nov 03 11:12:07.350377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.350347    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume\") pod \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\" (UID: \"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090\") "
Nov 03 11:12:07.350537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.350406    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-routing-overlay-volume\") pod \"cad4340e-0da6-4664-825f-cfbba8dc0017\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") "
Nov 03 11:12:07.350537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.350440    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials\") pod \"cad4340e-0da6-4664-825f-cfbba8dc0017\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") "
Nov 03 11:12:07.350778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.350753    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-kube-api-access-bpc42\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-kube-api-access-bpc42\") pod \"cad4340e-0da6-4664-825f-cfbba8dc0017\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") "
Nov 03 11:12:07.350984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.350962    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume\") pod \"cad4340e-0da6-4664-825f-cfbba8dc0017\" (UID: \"cad4340e-0da6-4664-825f-cfbba8dc0017\") "
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351189    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-certs\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351285    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406a01a7-6c91-4db3-a511-323ede863155-istiod-ca-cert\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351334    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-envoy\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351422    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-data\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351538    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-credential-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351634    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406a01a7-6c91-4db3-a511-323ede863155-istio-podinfo\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351755    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-istio-token\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351902    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfr8t\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-kube-api-access-tfr8t\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.351992    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.352159    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrxx\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-kube-api-access-kkrxx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.352178    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.352192    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:07.352471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.352440    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume" pod "cad4340e-0da6-4664-825f-cfbba8dc0017" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:12:07.353926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.353904    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-routing-overlay-volume" pod "cad4340e-0da6-4664-825f-cfbba8dc0017" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:12:07.354482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.354460    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume" pod "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" (UID: "12ed7c9c-5ca8-4974-9c3d-fff3c48fc090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:12:07.354800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.354782    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-kube-api-access-bpc42" pod "cad4340e-0da6-4664-825f-cfbba8dc0017" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017"). InnerVolumeSpecName "kube-api-access-bpc42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:07.357308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.357287    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials" pod "cad4340e-0da6-4664-825f-cfbba8dc0017" (UID: "cad4340e-0da6-4664-825f-cfbba8dc0017"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:07.370335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.370313    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9265bf-openshift-default-dockercfg-92kvj\""
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.453816    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-certs\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.453938    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406a01a7-6c91-4db3-a511-323ede863155-istiod-ca-cert\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.453988    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-envoy\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454081    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-data\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454135    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-credential-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454186    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406a01a7-6c91-4db3-a511-323ede863155-istio-podinfo\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454287    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-istio-token\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454335    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfr8t\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-kube-api-access-tfr8t\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454413    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454540    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpc42\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-kube-api-access-bpc42\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454558    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454594    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454609    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cad4340e-0da6-4664-825f-cfbba8dc0017-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.454623    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cad4340e-0da6-4664-825f-cfbba8dc0017-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.455040    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.455904    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-data\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.456202    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-certs\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.457604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.456897    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406a01a7-6c91-4db3-a511-323ede863155-istiod-ca-cert\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.466641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.464382    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-credential-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.478603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.475676    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406a01a7-6c91-4db3-a511-323ede863155-istio-podinfo\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.514410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.514378    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-envoy\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.515311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.515265    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfr8t\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-kube-api-access-tfr8t\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.549260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.548107    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-istio-token\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.634378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.634112    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:07.938614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.937623    5817 generic.go:356] "Generic (PLEG): container finished" podID="cad4340e-0da6-4664-825f-cfbba8dc0017" containerID="5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080" exitCode=0
Nov 03 11:12:07.938614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.938117    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"
Nov 03 11:12:07.939373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.938893    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" event={"ID":"cad4340e-0da6-4664-825f-cfbba8dc0017","Type":"ContainerDied","Data":"5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080"}
Nov 03 11:12:07.939373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.939117    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr" event={"ID":"cad4340e-0da6-4664-825f-cfbba8dc0017","Type":"ContainerDied","Data":"c85cbe3fce88c9b7beb997708b6ec528f85434ab655e0de02ae17b9af59a52d9"}
Nov 03 11:12:07.939373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.939155    5817 scope.go:122] "RemoveContainer" containerID="5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080"
Nov 03 11:12:07.949286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.943946    5817 generic.go:356] "Generic (PLEG): container finished" podID="12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" containerID="155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec" exitCode=0
Nov 03 11:12:07.949286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.944029    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"
Nov 03 11:12:07.949286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.944799    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" event={"ID":"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090","Type":"ContainerDied","Data":"155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec"}
Nov 03 11:12:07.949286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.944839    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w" event={"ID":"12ed7c9c-5ca8-4974-9c3d-fff3c48fc090","Type":"ContainerDied","Data":"92bdc30f4051bf273abf3059639cb2b207a17675023851d09a631b9ed0943c9e"}
Nov 03 11:12:07.967243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.967220    5817 scope.go:122] "RemoveContainer" containerID="5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080"
Nov 03 11:12:07.967929 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:07.967903    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080\": container with ID starting with 5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080 not found: ID does not exist" containerID="5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080"
Nov 03 11:12:07.968093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.968067    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080"} err="failed to get container status \"5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080\": rpc error: code = NotFound desc = could not find container \"5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080\": container with ID starting with 5e98e6fcb114490fedcb31bdc9cd1b2fe81e302bbb8476c8c6e3196694d52080 not found: ID does not exist"
Nov 03 11:12:07.968211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.968198    5817 scope.go:122] "RemoveContainer" containerID="155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec"
Nov 03 11:12:07.974532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.974510    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"]
Nov 03 11:12:07.981396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.981370    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-59ff9hrzr"]
Nov 03 11:12:07.988188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.988168    5817 scope.go:122] "RemoveContainer" containerID="155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec"
Nov 03 11:12:07.988477 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:07.988453    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec\": container with ID starting with 155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec not found: ID does not exist" containerID="155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec"
Nov 03 11:12:07.988540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.988498    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec"} err="failed to get container status \"155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec\": rpc error: code = NotFound desc = could not find container \"155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec\": container with ID starting with 155afc5d6d4921163e55de6b3f8fae4dda8d728d4ddbaeee8e3961eadec808ec not found: ID does not exist"
Nov 03 11:12:07.999018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:07.998992    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"]
Nov 03 11:12:08.015055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.015023    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6cd9nvx2w"]
Nov 03 11:12:08.180165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.159341    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:12:08.180165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.159404    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:12:08.180165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.159431    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:12:08.184554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.184516    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"]
Nov 03 11:12:08.444159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.444036    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:08.589960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.589923    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:08.956676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.956630    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" event={"ID":"406a01a7-6c91-4db3-a511-323ede863155","Type":"ContainerStarted","Data":"a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae"}
Nov 03 11:12:08.956911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.956699    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" event={"ID":"406a01a7-6c91-4db3-a511-323ede863155","Type":"ContainerStarted","Data":"0c780793abfe4f3bd5dc10324557a3935bce072dafec3c0b45371f101f3b36f8"}
Nov 03 11:12:08.980216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:08.980157    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" podStartSLOduration=1.980140994 podStartE2EDuration="1.980140994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:12:08.975562446 +0000 UTC m=+2559.181144497" watchObservedRunningTime="2026-11-03 11:12:08.980140994 +0000 UTC m=+2559.185723045"
Nov 03 11:12:09.043321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:09.043287    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:09.634787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:09.634754    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:09.641761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:09.641736    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:09.731603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:09.710834    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:09.906878 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:09.906792    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ed7c9c-5ca8-4974-9c3d-fff3c48fc090" path="/var/lib/kubelet/pods/12ed7c9c-5ca8-4974-9c3d-fff3c48fc090/volumes"
Nov 03 11:12:09.907414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:09.907389    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad4340e-0da6-4664-825f-cfbba8dc0017" path="/var/lib/kubelet/pods/cad4340e-0da6-4664-825f-cfbba8dc0017/volumes"
Nov 03 11:12:09.968161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:09.968131    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:09.969116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:09.969096    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:12:10.369612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:10.367816    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:10.999702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:10.997139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:11.701595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:11.694194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:12.371602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:12.363618    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:13.193735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:13.193603    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:13.762706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:13.761234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:14.413731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:14.413701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:15.268974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:15.268941    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"]
Nov 03 11:12:15.269407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:15.269180    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy" containerID="cri-o://852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc" gracePeriod=30
Nov 03 11:12:15.350244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:15.350210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:16.006600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:16.000053    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:16.721673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:16.707688    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:17.435654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:17.435603    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:18.161765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:18.161731    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:18.313744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:18.313705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:18.804890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:18.804858    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:19.029688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:19.029658    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"]
Nov 03 11:12:19.042646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:19.042616    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b917e05-7x9ts"]
Nov 03 11:12:19.369398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:19.369364    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:19.732322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:19.732275    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"]
Nov 03 11:12:19.732883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:19.732846    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy" containerID="cri-o://5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925" gracePeriod=30
Nov 03 11:12:19.799008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:19.798973    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:12:19.799204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:19.799046    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:12:19.909622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:19.909564    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1d31db-ff60-4822-b624-63cf8df3dce8" path="/var/lib/kubelet/pods/dc1d31db-ff60-4822-b624-63cf8df3dce8/volumes"
Nov 03 11:12:20.066700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.066511    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:20.583275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.583249    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 11:12:20.670324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.670235    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-certs\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.670490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.670338    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-istio-token\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-istio-token\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.670937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.670912    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-socket\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.671182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.670973    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2799609a-512e-4add-8320-979ebf65e473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799609a-512e-4add-8320-979ebf65e473-istio-podinfo\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.671182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.671008    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-envoy\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.671182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.671064    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-data\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.671567 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.671190    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2799609a-512e-4add-8320-979ebf65e473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799609a-512e-4add-8320-979ebf65e473-istiod-ca-cert\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.671567 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.671259    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-credential-socket\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.671567 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.671289    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-kube-api-access-5sxgb\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-kube-api-access-5sxgb\") pod \"2799609a-512e-4add-8320-979ebf65e473\" (UID: \"2799609a-512e-4add-8320-979ebf65e473\") "
Nov 03 11:12:20.671753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.671639    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-certs" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:20.671916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.671892    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-data" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:20.672087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.672061    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-socket" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:20.672154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.672135    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-credential-socket" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:20.672587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.672546    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.672684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.672597    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.672684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.672613    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.672684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.672627    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.672855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.672788    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2799609a-512e-4add-8320-979ebf65e473-istiod-ca-cert" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:12:20.674189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.674168    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-istio-token" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:20.674523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.674492    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-kube-api-access-5sxgb" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "kube-api-access-5sxgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:20.675343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.675315    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-envoy" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:20.675556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.675537    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2799609a-512e-4add-8320-979ebf65e473-istio-podinfo" pod "2799609a-512e-4add-8320-979ebf65e473" (UID: "2799609a-512e-4add-8320-979ebf65e473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:12:20.800638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.781556    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799609a-512e-4add-8320-979ebf65e473-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.800638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.781616    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxgb\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-kube-api-access-5sxgb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.800638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.781633    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799609a-512e-4add-8320-979ebf65e473-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.800638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.781647    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799609a-512e-4add-8320-979ebf65e473-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.800638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.781661    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799609a-512e-4add-8320-979ebf65e473-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:20.819170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:20.819095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:21.141408 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.141374    5817 generic.go:356] "Generic (PLEG): container finished" podID="2799609a-512e-4add-8320-979ebf65e473" containerID="852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc" exitCode=0
Nov 03 11:12:21.141619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.141420    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"
Nov 03 11:12:21.141619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.141461    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" event={"ID":"2799609a-512e-4add-8320-979ebf65e473","Type":"ContainerDied","Data":"852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc"}
Nov 03 11:12:21.141619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.141519    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9" event={"ID":"2799609a-512e-4add-8320-979ebf65e473","Type":"ContainerDied","Data":"da98ce08f3441385dcaef2f1d3cfec6cece8f00534c87cc395bc161a86aba043"}
Nov 03 11:12:21.141619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.141550    5817 scope.go:122] "RemoveContainer" containerID="852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc"
Nov 03 11:12:21.155606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.155563    5817 scope.go:122] "RemoveContainer" containerID="852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc"
Nov 03 11:12:21.155912 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:21.155891    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc\": container with ID starting with 852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc not found: ID does not exist" containerID="852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc"
Nov 03 11:12:21.156006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.155918    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc"} err="failed to get container status \"852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc\": rpc error: code = NotFound desc = could not find container \"852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc\": container with ID starting with 852fadec1e0d4282dab074fab6ecd93e2d2fb2591adab9bb7002cc67cabeafdc not found: ID does not exist"
Nov 03 11:12:21.170686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.170660    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"]
Nov 03 11:12:21.177233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.177209    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-6lxr9"]
Nov 03 11:12:21.280417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.280384    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:21.427166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.427082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:21.718635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.716868    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:21.906854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:21.906817    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2799609a-512e-4add-8320-979ebf65e473" path="/var/lib/kubelet/pods/2799609a-512e-4add-8320-979ebf65e473/volumes"
Nov 03 11:12:22.356295 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:22.356243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:22.479901 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:22.477596    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:12:22.512926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:22.512883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:22.741670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:22.728456    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:23.372617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:23.372540    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:24.067256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:24.067220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:24.780678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:24.780622    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:25.056726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.056703    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 11:12:25.064336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064311    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-data\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.064472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064391    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-certs\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.064525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064496    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-credential-socket\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.064594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064536    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istiod-ca-cert\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.064659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064605    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-data" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:25.064659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064626    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-token\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-token\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.064780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064678    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-certs" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:25.064780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064756    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-socket\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.064893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064790    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-kube-api-access-62xz4\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-kube-api-access-62xz4\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.064893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064853    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-credential-socket" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:25.064992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064920    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-podinfo\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.064992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064939    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istiod-ca-cert" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:12:25.065108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.065017    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-socket" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:25.065108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.064951    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-envoy\") pod \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\" (UID: \"257d4f7b-bf89-4f02-98f6-774fcd4994e3\") "
Nov 03 11:12:25.065837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.065816    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.065837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.065836    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.066025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.065846    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.066025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.065856    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.066025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.065864    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.067124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.067099    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-envoy" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:25.067311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.067295    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-podinfo" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:12:25.067686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.067545    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-token" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:25.067686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.067652    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-kube-api-access-62xz4" pod "257d4f7b-bf89-4f02-98f6-774fcd4994e3" (UID: "257d4f7b-bf89-4f02-98f6-774fcd4994e3"). InnerVolumeSpecName "kube-api-access-62xz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:25.168010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.167967    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62xz4\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-kube-api-access-62xz4\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.168010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.168007    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.168010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.168019    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.168375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.168027    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257d4f7b-bf89-4f02-98f6-774fcd4994e3-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:25.183148 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.183115    5817 generic.go:356] "Generic (PLEG): container finished" podID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerID="5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925" exitCode=0
Nov 03 11:12:25.183328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.183160    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"
Nov 03 11:12:25.183328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.183200    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" event={"ID":"257d4f7b-bf89-4f02-98f6-774fcd4994e3","Type":"ContainerDied","Data":"5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925"}
Nov 03 11:12:25.183328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.183255    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k" event={"ID":"257d4f7b-bf89-4f02-98f6-774fcd4994e3","Type":"ContainerDied","Data":"3694024a2e6796a87d0bcbadc8415a60c8222610e6ce0b0170c236c004f1d3ef"}
Nov 03 11:12:25.183328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.183284    5817 scope.go:122] "RemoveContainer" containerID="5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925"
Nov 03 11:12:25.196996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.196724    5817 scope.go:122] "RemoveContainer" containerID="5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925"
Nov 03 11:12:25.197183 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:25.197107    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925\": container with ID starting with 5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925 not found: ID does not exist" containerID="5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925"
Nov 03 11:12:25.197183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.197141    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925"} err="failed to get container status \"5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925\": rpc error: code = NotFound desc = could not find container \"5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925\": container with ID starting with 5f9db55f5dd2846c88281b7405ec410b7f473e90ad3e03edb0fb1463a3407925 not found: ID does not exist"
Nov 03 11:12:25.245138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.245098    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"]
Nov 03 11:12:25.250388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.250361    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-w2v4k"]
Nov 03 11:12:25.597766 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.597711    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:25.905196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:25.905114    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" path="/var/lib/kubelet/pods/257d4f7b-bf89-4f02-98f6-774fcd4994e3/volumes"
Nov 03 11:12:26.900596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:26.889063    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:27.691597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:27.682944    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:12:27.712606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:27.710166    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 11:12:27.717697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:27.717659    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:12:28.116649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:28.109550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:29.605813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:29.594044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:29.753779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:29.753733    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:12:29.753957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:29.753813    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:12:30.429657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:30.429620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:31.078237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:31.078197    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:31.701656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:31.697173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:32.368601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:32.366124    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:32.978362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:32.978326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:33.624154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:33.624116    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:34.096416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:34.096382    5817 scope.go:122] "RemoveContainer" containerID="b107e6a73adb514c32c693d030514336566534a88459d2c6a8ad7a2f27135f66"
Nov 03 11:12:34.112112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:34.112086    5817 scope.go:122] "RemoveContainer" containerID="e2cd4bd39b0311ae9b2d1dbfa6c08a0a02c2790006b67247c128c9c506259ea7"
Nov 03 11:12:34.125101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:34.125073    5817 scope.go:122] "RemoveContainer" containerID="e3737944727525ba5c3b86448c53c5cd9695569fd9fc63cbfb2635f54eb326fa"
Nov 03 11:12:34.136227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:34.136209    5817 scope.go:122] "RemoveContainer" containerID="01f2c2de50b7070ca10a1717efc93781eaa16cce93383d5a0c000cbd7bfb21e0"
Nov 03 11:12:34.268609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:34.266474    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:34.922558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:34.922516    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:35.687605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:35.684645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:36.342709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:36.342448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:36.944381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:36.944348    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:37.667896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:37.667861    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:38.392507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:38.392472    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:38.971765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:38.971728    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:39.686722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:39.684812    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:40.048325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.048223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:40.217931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.217881    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:40.640767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.640664    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:40.644615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.644553    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"]
Nov 03 11:12:40.647468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.647448    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy"
Nov 03 11:12:40.647663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.647642    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy"
Nov 03 11:12:40.647847 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.647835    5817 cpu_manager.go:414] "containerMap: removing container" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy"
Nov 03 11:12:40.647967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.647953    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy"
Nov 03 11:12:40.648273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.648258    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="257d4f7b-bf89-4f02-98f6-774fcd4994e3" containerName="istio-proxy"
Nov 03 11:12:40.648402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.648391    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2799609a-512e-4add-8320-979ebf65e473" containerName="istio-proxy"
Nov 03 11:12:40.656316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.656292    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"]
Nov 03 11:12:40.656428 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.656407    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.660767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.660748    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-951410f8ac99\""
Nov 03 11:12:40.663630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.661875    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fb5766\""
Nov 03 11:12:40.663630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.661995    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-e2e--b8bff1be-dockercfg-zvtrx\""
Nov 03 11:12:40.663630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.662177    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"routing-overlay\""
Nov 03 11:12:40.663630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.663005    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"kube-root-ca.crt\""
Nov 03 11:12:40.663630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.663225    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"openshift-service-ca.crt\""
Nov 03 11:12:40.816594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.807621    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.816594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.807716    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.816594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.807908    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.816594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.808054    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b92w4\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-kube-api-access-b92w4\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.918064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.911527    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.918064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.911628    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.918064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.911785    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.918064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.911907    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92w4\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-kube-api-access-b92w4\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.918064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.913049    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.918064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.913849    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.953844 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.946398    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92w4\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-kube-api-access-b92w4\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.958594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.954913    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:40.988537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:40.987706    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:41.366251 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:12:41.366220    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f90d72e_5f04_414d_95a2_0560d76f5c16.slice/crio-05f153d3b0576f26d5db398776200cc2afb8822b2b5cc20bbc01f0448aac99ca WatchSource:0}: Error finding container 05f153d3b0576f26d5db398776200cc2afb8822b2b5cc20bbc01f0448aac99ca: Status 404 returned error can't find the container with id 05f153d3b0576f26d5db398776200cc2afb8822b2b5cc20bbc01f0448aac99ca
Nov 03 11:12:41.367173 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:41.367154    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"]
Nov 03 11:12:41.451435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:41.451306    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:41.574054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:41.574016    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v" event={"ID":"6f90d72e-5f04-414d-95a2-0560d76f5c16","Type":"ContainerStarted","Data":"e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d"}
Nov 03 11:12:41.574235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:41.574081    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v" event={"ID":"6f90d72e-5f04-414d-95a2-0560d76f5c16","Type":"ContainerStarted","Data":"05f153d3b0576f26d5db398776200cc2afb8822b2b5cc20bbc01f0448aac99ca"}
Nov 03 11:12:41.574235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:41.574111    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:41.596313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:41.596266    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v" podStartSLOduration=1.596252703 podStartE2EDuration="1.596252703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:12:41.589006918 +0000 UTC m=+2591.794588969" watchObservedRunningTime="2026-11-03 11:12:41.596252703 +0000 UTC m=+2591.801834753"
Nov 03 11:12:42.153925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.153875    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"]
Nov 03 11:12:42.161379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.160660    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.176116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.176077    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"]
Nov 03 11:12:42.182622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.180229    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9265bf-serving-cert\""
Nov 03 11:12:42.247218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.247185    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:42.368459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.363853    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.368459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.363963    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-maas-api-tls\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.368459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.364201    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkx8j\" (UniqueName: \"kubernetes.io/projected/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-kube-api-access-gkx8j\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.368459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.364302    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-metrics-certs\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.396599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.390969    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:42.465907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.465860    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx8j\" (UniqueName: \"kubernetes.io/projected/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-kube-api-access-gkx8j\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.466097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.465974    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-metrics-certs\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.466097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.466032    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.466097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.466094    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-maas-api-tls\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.466841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.466813    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.468776 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.468756    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-maas-api-tls\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.468870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.468793    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-metrics-certs\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.474746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.474714    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx8j\" (UniqueName: \"kubernetes.io/projected/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-kube-api-access-gkx8j\") pod \"maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.511816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.511789    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:42.870636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.850388    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"]
Nov 03 11:12:42.912306 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:12:42.912257    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2abdf7_0efa_47ef_ba7c_70819dfbf167.slice/crio-2d4bdb33854850ba8a314e5a4334b8aae4d20c08480e1dc86307e7cc8bf2454b WatchSource:0}: Error finding container 2d4bdb33854850ba8a314e5a4334b8aae4d20c08480e1dc86307e7cc8bf2454b: Status 404 returned error can't find the container with id 2d4bdb33854850ba8a314e5a4334b8aae4d20c08480e1dc86307e7cc8bf2454b
Nov 03 11:12:42.922595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:42.916831    5817 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:12:43.242672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:43.239928    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:43.560123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:43.560095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:43.604800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:43.604771    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7" event={"ID":"7c2abdf7-0efa-47ef-ba7c-70819dfbf167","Type":"ContainerStarted","Data":"934fc27f52fc4d1a2c514f8e2917f406afe1b623b9a9196528834d3fbac6bc14"}
Nov 03 11:12:43.604951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:43.604832    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7" event={"ID":"7c2abdf7-0efa-47ef-ba7c-70819dfbf167","Type":"ContainerStarted","Data":"2d4bdb33854850ba8a314e5a4334b8aae4d20c08480e1dc86307e7cc8bf2454b"}
Nov 03 11:12:43.604951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:43.604861    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:43.622166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:43.622121    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7" podStartSLOduration=1.058724107 podStartE2EDuration="1.622108805s" totalImagesPullingTime="563.384698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:12:43.619695863 +0000 UTC m=+2593.825277938" watchObservedRunningTime="2026-11-03 11:12:43.622108805 +0000 UTC m=+2593.827690860"
Nov 03 11:12:44.016344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:44.014070    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:44.787225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:44.786807    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:45.313369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:45.313336    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:46.029167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:46.029134    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:46.862161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:46.862128    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:47.400943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:47.400908    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:48.208458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:48.208421    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:48.854537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:48.854502    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:49.450284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:49.450246    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:49.621820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:49.621793    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:12:50.070354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:50.070297    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:50.710750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:50.708326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:51.333141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:51.333104    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:52.006876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:52.006825    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:52.415131 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:12:52.415021    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:12:52.598082 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:52.592314    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:52.616645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:52.615854    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:12:53.375494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:53.375449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:53.964592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:53.964257    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:54.600599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:54.593283    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:55.297927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:55.297890    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:55.874604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:55.874412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:56.562353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:56.562155    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:57.164726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:57.164688    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:57.830603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:57.830345    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:58.600800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:58.600760    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:59.260916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:59.260877    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:12:59.701547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:12:59.701516    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:13:00.301593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:00.300085    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:01.327075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:01.327040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:02.410598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:02.408310    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:02.953783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:02.953735    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:03.225710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:03.225596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:03.856635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:03.855197    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:04.473736 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:04.473696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:05.115168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:05.115130    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:05.799598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:05.793724    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:06.391595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:06.390603    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:06.727009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:06.726975    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:07.062649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:07.062350    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:07.206598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:07.206194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:07.647935 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:07.647625    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"]
Nov 03 11:13:07.667842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:07.667812    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"]
Nov 03 11:13:07.668181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:07.668163    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:07.682658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:07.682634    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-60ad03-openshift-default-dockercfg-6q797\""
Nov 03 11:13:07.849596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:07.846016    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026034    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istiod-ca-cert\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026254    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-envoy\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026401    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgll\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-kube-api-access-ndgll\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026436    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-socket\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026467    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-podinfo\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026490    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-data\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026615    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-credential-socket\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026641    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-token\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.037169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.026710    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-certs\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.136614    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-certs\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.136751    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istiod-ca-cert\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.136952    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-envoy\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.137097    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgll\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-kube-api-access-ndgll\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.137143    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-socket\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.137190    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-podinfo\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.137223    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-data\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.137368    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-credential-socket\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.144966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.137396    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-token\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.158599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.149229    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:08.158599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.150986    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istiod-ca-cert\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.158599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.153562    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-data\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.158599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.154133    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-socket\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.158599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.157086    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-credential-socket\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.208323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.192166    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-podinfo\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.208323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.205311    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-certs\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.221919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.216296    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgll\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-kube-api-access-ndgll\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.221919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.216611    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-envoy\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.221919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.217247    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-token\") pod \"e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") " pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.275889 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.275859    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:08.426255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.425380    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:08.571865 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:13:08.571838    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b5ae23_6e01_43e1_a5e4_9a87c50edd64.slice/crio-969de8d813507c87941e3fa9834621608309abacccc8d66062873b90042ec827 WatchSource:0}: Error finding container 969de8d813507c87941e3fa9834621608309abacccc8d66062873b90042ec827: Status 404 returned error can't find the container with id 969de8d813507c87941e3fa9834621608309abacccc8d66062873b90042ec827
Nov 03 11:13:08.573349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.573326    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"]
Nov 03 11:13:08.575087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.575059    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:13:08.575153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.575122    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:13:08.575195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:08.575152    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:13:09.037846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:09.037814    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:09.139402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:09.139364    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" event={"ID":"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64","Type":"ContainerStarted","Data":"400e3df66e3806d2a15844a63c87f648c6fca1d31101706b6983758bcbd36c62"}
Nov 03 11:13:09.139641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:09.139417    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" event={"ID":"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64","Type":"ContainerStarted","Data":"969de8d813507c87941e3fa9834621608309abacccc8d66062873b90042ec827"}
Nov 03 11:13:09.167101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:09.167044    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podStartSLOduration=2.167026202 podStartE2EDuration="2.167026202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:09.157401931 +0000 UTC m=+2619.362983995" watchObservedRunningTime="2026-11-03 11:13:09.167026202 +0000 UTC m=+2619.372608256"
Nov 03 11:13:09.277982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:09.277949    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:09.339073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:09.333983    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:09.752862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:09.752821    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:10.150737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:10.150652    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:10.151897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:10.151875    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:13:10.655595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:10.648999    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:11.009771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:11.009725    5817 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:13:11.009982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:11.009798    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:13:11.378006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:11.377916    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:11.535936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:11.535892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:11.682497 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:11.682466    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:12.680170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.680131    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"]
Nov 03 11:13:12.684379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.684357    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:12.695203 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.695161    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-60ad03\""
Nov 03 11:13:12.697377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.697354    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"]
Nov 03 11:13:12.714070 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.713751    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:12.767476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.767439    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4\" (UID: \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:12.767943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.767907    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks976\" (UniqueName: \"kubernetes.io/projected/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-kube-api-access-ks976\") pod \"payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4\" (UID: \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:12.882605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.870935    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks976\" (UniqueName: \"kubernetes.io/projected/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-kube-api-access-ks976\") pod \"payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4\" (UID: \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:12.920913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.920883    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4\" (UID: \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:12.921653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.899706    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:12.922552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.922527    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4\" (UID: \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:12.926591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:12.924914    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks976\" (UniqueName: \"kubernetes.io/projected/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-kube-api-access-ks976\") pod \"payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4\" (UID: \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:13.070652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:13.048038    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:13.500201 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:13:13.500156    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode832b6ad_efdc_418a_8a5f_c04bbf2d8b09.slice/crio-a07d4930cf1c4aa5fef4ffae450154e2a4c3817d7d0f1943937f786f406e8a7b WatchSource:0}: Error finding container a07d4930cf1c4aa5fef4ffae450154e2a4c3817d7d0f1943937f786f406e8a7b: Status 404 returned error can't find the container with id a07d4930cf1c4aa5fef4ffae450154e2a4c3817d7d0f1943937f786f406e8a7b
Nov 03 11:13:13.502284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:13.502146    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"]
Nov 03 11:13:13.675594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:13.674778    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:14.212269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.212232    5817 generic.go:356] "Generic (PLEG): container finished" podID="406a01a7-6c91-4db3-a511-323ede863155" containerID="a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae" exitCode=0
Nov 03 11:13:14.212773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.212305    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" event={"ID":"406a01a7-6c91-4db3-a511-323ede863155","Type":"ContainerDied","Data":"a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae"}
Nov 03 11:13:14.212838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.212823    5817 scope.go:122] "RemoveContainer" containerID="a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae"
Nov 03 11:13:14.215648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.215548    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:13:14.215793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.215682    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:13:14.215793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.215697    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4" event={"ID":"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09","Type":"ContainerStarted","Data":"0116b0a7dbeab670717c57755588120707c00cf15c78dda485724befd897e7bb"}
Nov 03 11:13:14.215793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.215734    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:13:14.215793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.215746    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4" event={"ID":"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09","Type":"ContainerStarted","Data":"a07d4930cf1c4aa5fef4ffae450154e2a4c3817d7d0f1943937f786f406e8a7b"}
Nov 03 11:13:14.215793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.215776    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:14.255156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.255099    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4" podStartSLOduration=2.2550815330000002 podStartE2EDuration="2.255081533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:14.250477474 +0000 UTC m=+2624.456059527" watchObservedRunningTime="2026-11-03 11:13:14.255081533 +0000 UTC m=+2624.460663588"
Nov 03 11:13:14.323303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.323264    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:14.965597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:14.965371    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:15.232604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:15.232494    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" event={"ID":"406a01a7-6c91-4db3-a511-323ede863155","Type":"ContainerStarted","Data":"908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f"}
Nov 03 11:13:15.609657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:15.609050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:15.637595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:15.635586    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:13:15.637595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:15.635718    5817 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:13:15.637595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:15.635764    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 11:13:16.319903 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:16.319868    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:16.635125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:16.635041    5817 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:13:16.635125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:16.635094    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 11:13:16.929655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:16.927389    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:17.613612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:17.609139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:17.640595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:17.638205    5817 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:13:17.640595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:17.638281    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 11:13:18.249388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:18.249336    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:18.378200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:18.378162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:18.635347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:18.635248    5817 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:13:18.635347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:18.635333    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 11:13:19.009604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:19.009253    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:19.608051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:19.603690    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:19.642062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:19.638032    5817 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:13:19.642062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:19.638110    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 11:13:20.188598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:20.188506    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:20.640403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:20.640303    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:13:20.640775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:20.640752    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:13:20.641502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:20.641483    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:13:20.870274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:20.870239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:21.470596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:21.465694    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:22.168604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:22.158953    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:22.415253 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:22.415214    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:13:22.767445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:22.767412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:23.449839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:23.449787    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:24.201702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:24.201651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:24.873501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:24.870947    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:25.236646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:25.236615    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:13:25.624301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:25.624202    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:26.304595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:26.304276    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:26.964593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:26.964540    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:27.577640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:27.577601    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:28.235822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:28.231895    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:28.901611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:28.899339    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:29.548634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:29.548604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:29.758642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:29.758605    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:30.481449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:30.481396    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:31.193596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:31.187733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:31.874781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:31.874746    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:32.491713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:32.491678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:33.080649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:33.080610    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:33.752179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:33.751875    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:34.224982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:34.224952    5817 scope.go:122] "RemoveContainer" containerID="1d25debf0398de6d1144072e7a45dfe2892783d140a4cb7b38feb4b115545d90"
Nov 03 11:13:34.237279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:34.237258    5817 scope.go:122] "RemoveContainer" containerID="c4c71db5697a65b3fad646f0cb19b2e6e34c73d758dd9095151525e01e8ea214"
Nov 03 11:13:34.247869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:34.247838    5817 scope.go:122] "RemoveContainer" containerID="42bdb10f5319569bb6847d8ee644844fc569b80fc30c817db4c22ec7d02c2e3a"
Nov 03 11:13:34.258130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:34.258111    5817 scope.go:122] "RemoveContainer" containerID="af6836686633f903f19d6fb1e35960bb5d86b9564ed2bd6f6176457fad228c5b"
Nov 03 11:13:34.268931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:34.268906    5817 scope.go:122] "RemoveContainer" containerID="57134b167c76b24699560d01d1d5134be7dbb4770314249f3610b5ed93bb69cb"
Nov 03 11:13:34.475650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:34.468905    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:35.105407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:35.105371    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:35.772281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:35.772245    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:36.457815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:36.457778    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:38.203006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:38.202976    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:13:38.276647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:38.276613    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:38.413597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:38.410339    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:39.075472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:39.075441    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:39.870312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:39.870271    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:40.553626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:40.553589    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:41.254601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.251630    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:41.401149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.401116    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:41.882945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.872779    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"]
Nov 03 11:13:41.904864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.902683    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:41.917290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.911612    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"agc-cred-08c1ccc46862\""
Nov 03 11:13:41.924406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.921004    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"]
Nov 03 11:13:41.936101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.936074    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:41.936226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.936122    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:41.936289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.936247    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6xfj\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-kube-api-access-n6xfj\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:41.936394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:41.936381    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.037785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.037751    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6xfj\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-kube-api-access-n6xfj\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.037944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.037884    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.038051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.038031    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.038131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.038088    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.038643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.038624    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.038984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.038963    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.040907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.040882    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.045512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.045492    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6xfj\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-kube-api-access-n6xfj\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-574774c8g\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.236997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.236967    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.523645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.520130    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:42.588723 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:13:42.588692    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c73e58d_0605_4846_97ed_e7ddb710d6b5.slice/crio-6b7da0eb78779174af5ace52c7cda66fcb57c03b9db75d984e170b99abd66ffe WatchSource:0}: Error finding container 6b7da0eb78779174af5ace52c7cda66fcb57c03b9db75d984e170b99abd66ffe: Status 404 returned error can't find the container with id 6b7da0eb78779174af5ace52c7cda66fcb57c03b9db75d984e170b99abd66ffe
Nov 03 11:13:42.589710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.589689    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"]
Nov 03 11:13:42.756872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.756832    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" event={"ID":"3c73e58d-0605-4846-97ed-e7ddb710d6b5","Type":"ContainerStarted","Data":"b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f"}
Nov 03 11:13:42.757046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.756935    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:42.757046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.756960    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" event={"ID":"3c73e58d-0605-4846-97ed-e7ddb710d6b5","Type":"ContainerStarted","Data":"6b7da0eb78779174af5ace52c7cda66fcb57c03b9db75d984e170b99abd66ffe"}
Nov 03 11:13:42.776511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:42.776416    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" podStartSLOduration=1.7764040749999999 podStartE2EDuration="1.776404075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:42.774854148 +0000 UTC m=+2652.980436202" watchObservedRunningTime="2026-11-03 11:13:42.776404075 +0000 UTC m=+2652.981986175"
Nov 03 11:13:43.266298 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:43.266260    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:43.612649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:43.612356    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:43.626444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:43.626408    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"]
Nov 03 11:13:43.755598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:43.751044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:43.925760 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:43.925652    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-08c1ccc46862: secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:43.925983 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:43.925969    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g: secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:43.926165 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:43.926154    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials podName:3c73e58d-0605-4846-97ed-e7ddb710d6b5 nodeName:}" failed. No retries permitted until 2026-11-03 11:13:44.426107872 +0000 UTC m=+2654.631689904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" (UID: "3c73e58d-0605-4846-97ed-e7ddb710d6b5") : secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:44.225142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.225106    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"]
Nov 03 11:13:44.241604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.239493    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.241604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.241234    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"]
Nov 03 11:13:44.263596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.253798    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:13:44.263596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.253893    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:13:44.569238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.569152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:44.711598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.709277    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.711598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.709566    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb28p\" (UniqueName: \"kubernetes.io/projected/f05f4553-02e1-464a-8690-eee4089fad50-kube-api-access-lb28p\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.711598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.709649    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.711598 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:44.710821    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-08c1ccc46862: secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:44.711598 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:44.710856    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g: secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:44.711598 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:44.710971    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials podName:3c73e58d-0605-4846-97ed-e7ddb710d6b5 nodeName:}" failed. No retries permitted until 2026-11-03 11:13:45.710948151 +0000 UTC m=+2655.916530193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" (UID: "3c73e58d-0605-4846-97ed-e7ddb710d6b5") : secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:44.812086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.812049    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb28p\" (UniqueName: \"kubernetes.io/projected/f05f4553-02e1-464a-8690-eee4089fad50-kube-api-access-lb28p\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.812277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.812108    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.812349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.812324    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.813035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.813011    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.814697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.814678    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.820415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.820351    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb28p\" (UniqueName: \"kubernetes.io/projected/f05f4553-02e1-464a-8690-eee4089fad50-kube-api-access-lb28p\") pod \"maas-api-revoke-sub-dea9d6ee-l264b\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:44.866188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:44.866162    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:45.034595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.034097    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"]
Nov 03 11:13:45.046593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.046060    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"]
Nov 03 11:13:45.056108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.056074    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3625d-r9nq2"]
Nov 03 11:13:45.061922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.061893    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dc983c0-4d298"]
Nov 03 11:13:45.158786 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:13:45.158757    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05f4553_02e1_464a_8690_eee4089fad50.slice/crio-eadf7376b1f9b1710961a6ce96d3d2f825333be28107ba7d5016ac29bc8f11c5 WatchSource:0}: Error finding container eadf7376b1f9b1710961a6ce96d3d2f825333be28107ba7d5016ac29bc8f11c5: Status 404 returned error can't find the container with id eadf7376b1f9b1710961a6ce96d3d2f825333be28107ba7d5016ac29bc8f11c5
Nov 03 11:13:45.158983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.158963    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"]
Nov 03 11:13:45.410683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.406388    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:45.727099 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:45.727074    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-08c1ccc46862: secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:45.727529 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:45.727107    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g: secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:45.727529 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:45.727171    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials podName:3c73e58d-0605-4846-97ed-e7ddb710d6b5 nodeName:}" failed. No retries permitted until 2026-11-03 11:13:47.727152977 +0000 UTC m=+2657.932735009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" (UID: "3c73e58d-0605-4846-97ed-e7ddb710d6b5") : secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:45.876758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.876720    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b" event={"ID":"f05f4553-02e1-464a-8690-eee4089fad50","Type":"ContainerStarted","Data":"d213f819ad91b4303d082a4c85b23887c4b7e5333391192ae571b65713ab8408"}
Nov 03 11:13:45.876927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.876779    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b" event={"ID":"f05f4553-02e1-464a-8690-eee4089fad50","Type":"ContainerStarted","Data":"eadf7376b1f9b1710961a6ce96d3d2f825333be28107ba7d5016ac29bc8f11c5"}
Nov 03 11:13:45.895884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.895830    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b" podStartSLOduration=1.895813381 podStartE2EDuration="1.895813381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:45.892196458 +0000 UTC m=+2656.097778511" watchObservedRunningTime="2026-11-03 11:13:45.895813381 +0000 UTC m=+2656.101395432"
Nov 03 11:13:45.910641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.910612    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b6fd3b-49e4-48ef-bce0-3c758e15dbe6" path="/var/lib/kubelet/pods/09b6fd3b-49e4-48ef-bce0-3c758e15dbe6/volumes"
Nov 03 11:13:45.911170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:45.911154    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613437d2-207b-4fad-9300-f594fb5cba7f" path="/var/lib/kubelet/pods/613437d2-207b-4fad-9300-f594fb5cba7f/volumes"
Nov 03 11:13:46.080545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:46.080449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:46.718603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:46.717710    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:46.888102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:46.888062    5817 generic.go:356] "Generic (PLEG): container finished" podID="f05f4553-02e1-464a-8690-eee4089fad50" containerID="d213f819ad91b4303d082a4c85b23887c4b7e5333391192ae571b65713ab8408" exitCode=0
Nov 03 11:13:46.888627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:46.888150    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b" event={"ID":"f05f4553-02e1-464a-8690-eee4089fad50","Type":"ContainerDied","Data":"d213f819ad91b4303d082a4c85b23887c4b7e5333391192ae571b65713ab8408"}
Nov 03 11:13:47.311600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:47.307379    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:47.765527 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:47.765497    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-08c1ccc46862: secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:47.765527 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:47.765526    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g: secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:47.765771 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:47.765611    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials podName:3c73e58d-0605-4846-97ed-e7ddb710d6b5 nodeName:}" failed. No retries permitted until 2026-11-03 11:13:51.765562606 +0000 UTC m=+2661.971144648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" (UID: "3c73e58d-0605-4846-97ed-e7ddb710d6b5") : secret "agc-cred-08c1ccc46862" not found
Nov 03 11:13:47.806255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:47.806221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:48.152464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.152441    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:48.253361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.253330    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-xks-ca\") pod \"f05f4553-02e1-464a-8690-eee4089fad50\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") "
Nov 03 11:13:48.253536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.253375    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05f4553-02e1-464a-8690-eee4089fad50-kube-api-access-lb28p\" (UniqueName: \"kubernetes.io/projected/f05f4553-02e1-464a-8690-eee4089fad50-kube-api-access-lb28p\") pod \"f05f4553-02e1-464a-8690-eee4089fad50\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") "
Nov 03 11:13:48.253621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.253562    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-openshift-ca\") pod \"f05f4553-02e1-464a-8690-eee4089fad50\" (UID: \"f05f4553-02e1-464a-8690-eee4089fad50\") "
Nov 03 11:13:48.253931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.253901    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-openshift-ca" pod "f05f4553-02e1-464a-8690-eee4089fad50" (UID: "f05f4553-02e1-464a-8690-eee4089fad50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:13:48.254309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.254291    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:48.255786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.255760    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05f4553-02e1-464a-8690-eee4089fad50-kube-api-access-lb28p" pod "f05f4553-02e1-464a-8690-eee4089fad50" (UID: "f05f4553-02e1-464a-8690-eee4089fad50"). InnerVolumeSpecName "kube-api-access-lb28p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:13:48.255900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.255817    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-xks-ca" pod "f05f4553-02e1-464a-8690-eee4089fad50" (UID: "f05f4553-02e1-464a-8690-eee4089fad50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:13:48.356019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.355991    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05f4553-02e1-464a-8690-eee4089fad50-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:48.356019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.356017    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb28p\" (UniqueName: \"kubernetes.io/projected/f05f4553-02e1-464a-8690-eee4089fad50-kube-api-access-lb28p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:48.910881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.910845    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"
Nov 03 11:13:48.910881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.910861    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b" event={"ID":"f05f4553-02e1-464a-8690-eee4089fad50","Type":"ContainerDied","Data":"eadf7376b1f9b1710961a6ce96d3d2f825333be28107ba7d5016ac29bc8f11c5"}
Nov 03 11:13:48.911105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:48.910914    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eadf7376b1f9b1710961a6ce96d3d2f825333be28107ba7d5016ac29bc8f11c5"
Nov 03 11:13:49.009804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:49.009769    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:49.160471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:49.160436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:49.329810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:49.329774    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:49.869817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:49.869736    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" podUID="3c73e58d-0605-4846-97ed-e7ddb710d6b5" containerName="payload-processing" containerID="cri-o://b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f" gracePeriod=25
Nov 03 11:13:49.918598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:49.918371    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:50.224819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.224785    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:50.299194 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.299162    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials\") pod \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") "
Nov 03 11:13:50.299509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.299486    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-routing-overlay-volume\") pod \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") "
Nov 03 11:13:50.299623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.299537    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-plugins-config-volume\") pod \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") "
Nov 03 11:13:50.299701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.299669    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-kube-api-access-n6xfj\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-kube-api-access-n6xfj\") pod \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\" (UID: \"3c73e58d-0605-4846-97ed-e7ddb710d6b5\") "
Nov 03 11:13:50.299934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.299912    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-routing-overlay-volume" pod "3c73e58d-0605-4846-97ed-e7ddb710d6b5" (UID: "3c73e58d-0605-4846-97ed-e7ddb710d6b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:13:50.300001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.299955    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-plugins-config-volume" pod "3c73e58d-0605-4846-97ed-e7ddb710d6b5" (UID: "3c73e58d-0605-4846-97ed-e7ddb710d6b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:13:50.300365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.300348    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:50.300435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.300368    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73e58d-0605-4846-97ed-e7ddb710d6b5-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:50.301942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.301919    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-kube-api-access-n6xfj" pod "3c73e58d-0605-4846-97ed-e7ddb710d6b5" (UID: "3c73e58d-0605-4846-97ed-e7ddb710d6b5"). InnerVolumeSpecName "kube-api-access-n6xfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:13:50.302059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.301985    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials" pod "3c73e58d-0605-4846-97ed-e7ddb710d6b5" (UID: "3c73e58d-0605-4846-97ed-e7ddb710d6b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:13:50.402467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.402432    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6xfj\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-kube-api-access-n6xfj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:50.402467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.402463    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c73e58d-0605-4846-97ed-e7ddb710d6b5-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:50.517136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:50.516082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:51.074172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.074132    5817 generic.go:356] "Generic (PLEG): container finished" podID="3c73e58d-0605-4846-97ed-e7ddb710d6b5" containerID="b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f" exitCode=0
Nov 03 11:13:51.074489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.074464    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" event={"ID":"3c73e58d-0605-4846-97ed-e7ddb710d6b5","Type":"ContainerDied","Data":"b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f"}
Nov 03 11:13:51.074733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.074693    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g" event={"ID":"3c73e58d-0605-4846-97ed-e7ddb710d6b5","Type":"ContainerDied","Data":"6b7da0eb78779174af5ace52c7cda66fcb57c03b9db75d984e170b99abd66ffe"}
Nov 03 11:13:51.075086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.075070    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"
Nov 03 11:13:51.090028 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.090003    5817 scope.go:122] "RemoveContainer" containerID="b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f"
Nov 03 11:13:51.237595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.234813    5817 scope.go:122] "RemoveContainer" containerID="b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f"
Nov 03 11:13:51.237595 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:51.235558    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f\": container with ID starting with b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f not found: ID does not exist" containerID="b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f"
Nov 03 11:13:51.237595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.235610    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f"} err="failed to get container status \"b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f\": rpc error: code = NotFound desc = could not find container \"b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f\": container with ID starting with b6f6091fd400ca28d635251d376731051f203ebd59dd20c7d7f33290b6386e6f not found: ID does not exist"
Nov 03 11:13:51.264596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.256598    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"]
Nov 03 11:13:51.297832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.292685    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-574774c8g"]
Nov 03 11:13:51.437216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.435263    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:51.905983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:51.905899    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c73e58d-0605-4846-97ed-e7ddb710d6b5" path="/var/lib/kubelet/pods/3c73e58d-0605-4846-97ed-e7ddb710d6b5/volumes"
Nov 03 11:13:52.011833 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:52.011802    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:52.192883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:52.192425    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:52.418340 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:13:52.418303    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:13:52.935561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:52.935517    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:53.405308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:53.405225    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:53.974339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:53.974304    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:54.670627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.667966    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:54.753621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.753460    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"]
Nov 03 11:13:54.754929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.754898    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f05f4553-02e1-464a-8690-eee4089fad50" containerName="revoke-keys"
Nov 03 11:13:54.754929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.754927    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05f4553-02e1-464a-8690-eee4089fad50" containerName="revoke-keys"
Nov 03 11:13:54.755151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.755001    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3c73e58d-0605-4846-97ed-e7ddb710d6b5" containerName="payload-processing"
Nov 03 11:13:54.755151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.755008    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c73e58d-0605-4846-97ed-e7ddb710d6b5" containerName="payload-processing"
Nov 03 11:13:54.755282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.755183    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c73e58d-0605-4846-97ed-e7ddb710d6b5" containerName="payload-processing"
Nov 03 11:13:54.755282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.755209    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05f4553-02e1-464a-8690-eee4089fad50" containerName="revoke-keys"
Nov 03 11:13:54.764091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.764071    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"]
Nov 03 11:13:54.764205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.764157    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:54.770379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.770359    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:13:54.770516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.770407    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:13:54.862135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.862102    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:54.862290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.862270    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:54.862462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.862448    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjm8\" (UniqueName: \"kubernetes.io/projected/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-kube-api-access-6sjm8\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:54.985834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.985801    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:55.007601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.997796    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:55.007601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:54.998127    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjm8\" (UniqueName: \"kubernetes.io/projected/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-kube-api-access-6sjm8\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:55.007601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:55.003215    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:55.007601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:55.004936    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:55.007601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:55.007061    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjm8\" (UniqueName: \"kubernetes.io/projected/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-kube-api-access-6sjm8\") pod \"maas-api-revoke-sub-43d197c3-8st2f\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:55.083184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:55.083155    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:55.381261 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:13:55.381231    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0f2b24_bed5_4978_aa5d_a21bfd6b7117.slice/crio-87a24cfa779c8f08cefd70ea9dbf2acb9ab42fa7e12c4e148f3f410de030ad1a WatchSource:0}: Error finding container 87a24cfa779c8f08cefd70ea9dbf2acb9ab42fa7e12c4e148f3f410de030ad1a: Status 404 returned error can't find the container with id 87a24cfa779c8f08cefd70ea9dbf2acb9ab42fa7e12c4e148f3f410de030ad1a
Nov 03 11:13:55.382955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:55.382929    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"]
Nov 03 11:13:55.445642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:55.443963    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:56.116661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:56.116541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:56.150286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:56.150255    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f" event={"ID":"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117","Type":"ContainerStarted","Data":"cadd7d99abcf9273708d70fceec95c143ba5d5966e2d4adeea911567fbc7e140"}
Nov 03 11:13:56.150447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:56.150308    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f" event={"ID":"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117","Type":"ContainerStarted","Data":"87a24cfa779c8f08cefd70ea9dbf2acb9ab42fa7e12c4e148f3f410de030ad1a"}
Nov 03 11:13:56.169388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:56.169346    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f" podStartSLOduration=2.169332464 podStartE2EDuration="2.169332464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:56.164764684 +0000 UTC m=+2666.370346746" watchObservedRunningTime="2026-11-03 11:13:56.169332464 +0000 UTC m=+2666.374914515"
Nov 03 11:13:56.755598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:56.753724    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:57.161213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:57.161127    5817 generic.go:356] "Generic (PLEG): container finished" podID="1b0f2b24-bed5-4978-aa5d-a21bfd6b7117" containerID="cadd7d99abcf9273708d70fceec95c143ba5d5966e2d4adeea911567fbc7e140" exitCode=0
Nov 03 11:13:57.161637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:57.161211    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f" event={"ID":"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117","Type":"ContainerDied","Data":"cadd7d99abcf9273708d70fceec95c143ba5d5966e2d4adeea911567fbc7e140"}
Nov 03 11:13:57.444188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:57.444151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:58.204977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.204935    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:58.473526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.473459    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:58.580663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.580625    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-xks-ca\") pod \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") "
Nov 03 11:13:58.580862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.580706    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-openshift-ca\") pod \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") "
Nov 03 11:13:58.580977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.580965    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-kube-api-access-6sjm8\" (UniqueName: \"kubernetes.io/projected/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-kube-api-access-6sjm8\") pod \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\" (UID: \"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117\") "
Nov 03 11:13:58.581098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.581075    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-openshift-ca" pod "1b0f2b24-bed5-4978-aa5d-a21bfd6b7117" (UID: "1b0f2b24-bed5-4978-aa5d-a21bfd6b7117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:13:58.581743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.581723    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:58.583186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.583157    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-xks-ca" pod "1b0f2b24-bed5-4978-aa5d-a21bfd6b7117" (UID: "1b0f2b24-bed5-4978-aa5d-a21bfd6b7117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:13:58.583285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.583217    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-kube-api-access-6sjm8" pod "1b0f2b24-bed5-4978-aa5d-a21bfd6b7117" (UID: "1b0f2b24-bed5-4978-aa5d-a21bfd6b7117"). InnerVolumeSpecName "kube-api-access-6sjm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:13:58.684282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.684246    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:58.684282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.684284    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjm8\" (UniqueName: \"kubernetes.io/projected/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117-kube-api-access-6sjm8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:58.858659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:58.855169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:13:59.235350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:59.235319    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"
Nov 03 11:13:59.236656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:59.235382    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f" event={"ID":"1b0f2b24-bed5-4978-aa5d-a21bfd6b7117","Type":"ContainerDied","Data":"87a24cfa779c8f08cefd70ea9dbf2acb9ab42fa7e12c4e148f3f410de030ad1a"}
Nov 03 11:13:59.236802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:59.236688    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87a24cfa779c8f08cefd70ea9dbf2acb9ab42fa7e12c4e148f3f410de030ad1a"
Nov 03 11:13:59.516135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:13:59.516047    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:00.276422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:00.276382    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:01.415284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:01.413689    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:02.246896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:02.246862    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:03.409600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:03.406533    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:04.059419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:04.059386    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:04.754606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:04.739890    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:05.460370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:05.460332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:05.739869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:05.739775    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:05.879778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:05.879738    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:06.026304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.026230    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:06.046249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.046218    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"]
Nov 03 11:14:06.047156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.047137    5817 cpu_manager.go:414] "containerMap: removing container" podUID="1b0f2b24-bed5-4978-aa5d-a21bfd6b7117" containerName="revoke-keys"
Nov 03 11:14:06.047156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.047155    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0f2b24-bed5-4978-aa5d-a21bfd6b7117" containerName="revoke-keys"
Nov 03 11:14:06.047386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.047375    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0f2b24-bed5-4978-aa5d-a21bfd6b7117" containerName="revoke-keys"
Nov 03 11:14:06.051620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.051548    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.062145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.056544    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:14:06.062145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.056564    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:14:06.073664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.073641    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"]
Nov 03 11:14:06.084795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.084770    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glph\" (UniqueName: \"kubernetes.io/projected/2cfd3202-6582-471c-be21-211b5ace4aee-kube-api-access-2glph\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.084901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.084831    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.084901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.084868    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.186552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.186515    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glph\" (UniqueName: \"kubernetes.io/projected/2cfd3202-6582-471c-be21-211b5ace4aee-kube-api-access-2glph\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.186790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.186608    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.186790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.186662    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.208875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.208805    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.213597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.213346    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.249592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.248239    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glph\" (UniqueName: \"kubernetes.io/projected/2cfd3202-6582-471c-be21-211b5ace4aee-kube-api-access-2glph\") pod \"maas-api-revoke-sub-fd93a8ab-nkmr9\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.385657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.380324    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:06.764270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.764240    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"]
Nov 03 11:14:06.808796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.808763    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"]
Nov 03 11:14:06.809095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.809079    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:06.823639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.821402    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"agc-cred-4cb90d41b019\""
Nov 03 11:14:06.823639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.821528    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-e2e--b43e39c2-dockercfg-mwn6c\""
Nov 03 11:14:06.824269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.824248    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"openshift-service-ca.crt\""
Nov 03 11:14:06.824517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.824500    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"kube-root-ca.crt\""
Nov 03 11:14:06.825325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.825305    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"routing-overlay\""
Nov 03 11:14:06.825552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.825534    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9f53bb\""
Nov 03 11:14:06.825695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.825640    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"agc-cred-2bb5b017fffc\""
Nov 03 11:14:06.855421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:06.854511    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:07.034048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.026980    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:07.062381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.041838    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.070660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.062709    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.070951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.070921    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664d6\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-kube-api-access-664d6\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.082279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.071287    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.192382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.191832    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.192382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.192151    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.192382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.192202    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.192382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.192230    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664d6\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-kube-api-access-664d6\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.206657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.193646    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.278595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.266927    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.316647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.309254    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.376593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.370217    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-664d6\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-kube-api-access-664d6\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.480598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.480143    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:07.684267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.683223    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"]
Nov 03 11:14:07.846407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:07.846372    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:08.090599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.086863    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9" event={"ID":"2cfd3202-6582-471c-be21-211b5ace4aee","Type":"ContainerStarted","Data":"fc0009422e231814f95dc13ff1f57234f292d8cea7f7e9f457eecba5b5790d96"}
Nov 03 11:14:08.126597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.124713    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"]
Nov 03 11:14:08.132136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.132107    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"]
Nov 03 11:14:08.132285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.132156    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"]
Nov 03 11:14:08.134620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.134555    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.141714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.139554    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"]
Nov 03 11:14:08.141714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.139695    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.147625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.147604    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"payload-processing-external-model-e2e--f843bc49-dockercfg-ggvq5\""
Nov 03 11:14:08.147715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.147676    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"agc-cred-540cdc5a672e\""
Nov 03 11:14:08.329217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.328303    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n7hf\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-kube-api-access-5n7hf\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.329217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.328401    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.329217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.328503    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8sqw\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-kube-api-access-h8sqw\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.329217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.328590    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.329217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.328632    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.329217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.328693    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.329217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.328724    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.329217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.328874    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.379095 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:14:08.379070    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24ebefd_9977_4a72_8103_9cacaf5d518c.slice/crio-040a625bfce01f6c7fbddb97c1840418a675efaefb71a8b7ba6b1a0ead562f1d WatchSource:0}: Error finding container 040a625bfce01f6c7fbddb97c1840418a675efaefb71a8b7ba6b1a0ead562f1d: Status 404 returned error can't find the container with id 040a625bfce01f6c7fbddb97c1840418a675efaefb71a8b7ba6b1a0ead562f1d
Nov 03 11:14:08.383021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.382993    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"]
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.430179    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8sqw\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-kube-api-access-h8sqw\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.430249    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.430294    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.430345    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.430365    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.430465    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.430550    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n7hf\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-kube-api-access-5n7hf\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.430660    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.431444    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:08.431479    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-a1416845a683: secret "agc-cred-a1416845a683" not found
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:08.431502    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq: secret "agc-cred-a1416845a683" not found
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:08.431555    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials podName:181266b1-4afa-4379-8460-3a480576d67e nodeName:}" failed. No retries permitted until 2026-11-03 11:14:08.931536837 +0000 UTC m=+2679.137118880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq" (UID: "181266b1-4afa-4379-8460-3a480576d67e") : secret "agc-cred-a1416845a683" not found
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.432118    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.431486    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.433623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.432792    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.435925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.435904    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.441369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.441348    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n7hf\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-kube-api-access-5n7hf\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.441441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.441388    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8sqw\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-kube-api-access-h8sqw\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:08.472218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.472185    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"]
Nov 03 11:14:08.474229 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:08.474196    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq" podUID="181266b1-4afa-4379-8460-3a480576d67e"
Nov 03 11:14:08.480364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.480340    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"]
Nov 03 11:14:08.482876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.482464    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:08.583224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.583113    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:08.811621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:08.805604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:09.007550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.007321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:09.037596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.025746    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:09.103616 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:09.039720    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-a1416845a683: secret "agc-cred-a1416845a683" not found
Nov 03 11:14:09.150240 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:09.113643    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq: secret "agc-cred-a1416845a683" not found
Nov 03 11:14:09.150240 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:09.113931    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials podName:181266b1-4afa-4379-8460-3a480576d67e nodeName:}" failed. No retries permitted until 2026-11-03 11:14:10.113896786 +0000 UTC m=+2680.319478835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq" (UID: "181266b1-4afa-4379-8460-3a480576d67e") : secret "agc-cred-a1416845a683" not found
Nov 03 11:14:09.555120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.555086    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"]
Nov 03 11:14:09.559192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.558077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:09.574235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.574206    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7" event={"ID":"b24ebefd-9977-4a72-8103-9cacaf5d518c","Type":"ContainerStarted","Data":"92fbda87e394a01efbc5757774021c6afc675e9c8115292ef605afde172bcce6"}
Nov 03 11:14:09.574432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.574418    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:09.575478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.575446    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7" event={"ID":"b24ebefd-9977-4a72-8103-9cacaf5d518c","Type":"ContainerStarted","Data":"040a625bfce01f6c7fbddb97c1840418a675efaefb71a8b7ba6b1a0ead562f1d"}
Nov 03 11:14:09.577130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.576408    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"]
Nov 03 11:14:09.582155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.582135    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:09.583134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.583107    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9" event={"ID":"2cfd3202-6582-471c-be21-211b5ace4aee","Type":"ContainerStarted","Data":"5ab6d9675bbce4cb9e1730e6a94362e37aed5a07f91d4db01a0f9141c7d3a3b4"}
Nov 03 11:14:09.625894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.618736    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:09.639533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.639475    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7" podStartSLOduration=3.639457416 podStartE2EDuration="3.639457416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:09.599201546 +0000 UTC m=+2679.804783601" watchObservedRunningTime="2026-11-03 11:14:09.639457416 +0000 UTC m=+2679.845039465"
Nov 03 11:14:09.772478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.772455    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-kube-api-access-h8sqw\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-kube-api-access-h8sqw\") pod \"181266b1-4afa-4379-8460-3a480576d67e\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") "
Nov 03 11:14:09.772691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.772673    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-routing-overlay-volume\") pod \"181266b1-4afa-4379-8460-3a480576d67e\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") "
Nov 03 11:14:09.772907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.772892    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-plugins-config-volume\") pod \"181266b1-4afa-4379-8460-3a480576d67e\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") "
Nov 03 11:14:09.773001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.772928    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-routing-overlay-volume" pod "181266b1-4afa-4379-8460-3a480576d67e" (UID: "181266b1-4afa-4379-8460-3a480576d67e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:09.773342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.773316    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-plugins-config-volume" pod "181266b1-4afa-4379-8460-3a480576d67e" (UID: "181266b1-4afa-4379-8460-3a480576d67e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:09.773716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.773693    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:09.773814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.773722    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181266b1-4afa-4379-8460-3a480576d67e-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:09.774684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.774662    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-kube-api-access-h8sqw" pod "181266b1-4afa-4379-8460-3a480576d67e" (UID: "181266b1-4afa-4379-8460-3a480576d67e"). InnerVolumeSpecName "kube-api-access-h8sqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:09.875304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:09.875270    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8sqw\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-kube-api-access-h8sqw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:10.124597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.124502    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq\" (UID: \"181266b1-4afa-4379-8460-3a480576d67e\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:10.125075 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:10.124821    5817 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-a1416845a683: secret "agc-cred-a1416845a683" not found
Nov 03 11:14:10.125075 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:10.124839    5817 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq: secret "agc-cred-a1416845a683" not found
Nov 03 11:14:10.125075 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:10.124898    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials podName:181266b1-4afa-4379-8460-3a480576d67e nodeName:}" failed. No retries permitted until 2026-11-03 11:14:12.124879456 +0000 UTC m=+2682.330461507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq" (UID: "181266b1-4afa-4379-8460-3a480576d67e") : secret "agc-cred-a1416845a683" not found
Nov 03 11:14:10.127401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.127171    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:10.597536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.597432    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll" event={"ID":"7dafcc8a-f9aa-433f-a048-5033734fa494","Type":"ContainerStarted","Data":"9a1706a5be60e20031d3a8621dcc5fc1b61d0248a05ea9500425ace076ff29d0"}
Nov 03 11:14:10.597536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.597498    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll" event={"ID":"7dafcc8a-f9aa-433f-a048-5033734fa494","Type":"ContainerStarted","Data":"4abe2ecde22335b69cb28cf1897c6290ceddd5cc494f0821113e208dd83ca680"}
Nov 03 11:14:10.597787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.597540    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:10.599498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.599474    5817 generic.go:356] "Generic (PLEG): container finished" podID="2cfd3202-6582-471c-be21-211b5ace4aee" containerID="5ab6d9675bbce4cb9e1730e6a94362e37aed5a07f91d4db01a0f9141c7d3a3b4" exitCode=0
Nov 03 11:14:10.599682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.599554    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9" event={"ID":"2cfd3202-6582-471c-be21-211b5ace4aee","Type":"ContainerDied","Data":"5ab6d9675bbce4cb9e1730e6a94362e37aed5a07f91d4db01a0f9141c7d3a3b4"}
Nov 03 11:14:10.599682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.599663    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"
Nov 03 11:14:10.617648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.617597    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll" podStartSLOduration=3.617562769 podStartE2EDuration="3.617562769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:10.613289584 +0000 UTC m=+2680.818871635" watchObservedRunningTime="2026-11-03 11:14:10.617562769 +0000 UTC m=+2680.823144821"
Nov 03 11:14:10.648872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.648830    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"]
Nov 03 11:14:10.654132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.654102    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6855rd7dq"]
Nov 03 11:14:10.769514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.769480    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181266b1-4afa-4379-8460-3a480576d67e-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:10.777834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.776588    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:10.899793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.899758    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:10.973405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.973369    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-openshift-ca\") pod \"2cfd3202-6582-471c-be21-211b5ace4aee\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") "
Nov 03 11:14:10.973628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.973587    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-xks-ca\") pod \"2cfd3202-6582-471c-be21-211b5ace4aee\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") "
Nov 03 11:14:10.973797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.973775    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-openshift-ca" pod "2cfd3202-6582-471c-be21-211b5ace4aee" (UID: "2cfd3202-6582-471c-be21-211b5ace4aee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:10.973928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.973910    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfd3202-6582-471c-be21-211b5ace4aee-kube-api-access-2glph\" (UniqueName: \"kubernetes.io/projected/2cfd3202-6582-471c-be21-211b5ace4aee-kube-api-access-2glph\") pod \"2cfd3202-6582-471c-be21-211b5ace4aee\" (UID: \"2cfd3202-6582-471c-be21-211b5ace4aee\") "
Nov 03 11:14:10.974610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.974549    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:10.975838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.975811    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-xks-ca" pod "2cfd3202-6582-471c-be21-211b5ace4aee" (UID: "2cfd3202-6582-471c-be21-211b5ace4aee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:14:10.976356 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:10.976336    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfd3202-6582-471c-be21-211b5ace4aee-kube-api-access-2glph" pod "2cfd3202-6582-471c-be21-211b5ace4aee" (UID: "2cfd3202-6582-471c-be21-211b5ace4aee"). InnerVolumeSpecName "kube-api-access-2glph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:11.076101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:11.076067    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfd3202-6582-471c-be21-211b5ace4aee-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:11.076101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:11.076100    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2glph\" (UniqueName: \"kubernetes.io/projected/2cfd3202-6582-471c-be21-211b5ace4aee-kube-api-access-2glph\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:11.454596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:11.452497    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:11.614274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:11.614244    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"
Nov 03 11:14:11.614467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:11.614269    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9" event={"ID":"2cfd3202-6582-471c-be21-211b5ace4aee","Type":"ContainerDied","Data":"fc0009422e231814f95dc13ff1f57234f292d8cea7f7e9f457eecba5b5790d96"}
Nov 03 11:14:11.614467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:11.614320    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0009422e231814f95dc13ff1f57234f292d8cea7f7e9f457eecba5b5790d96"
Nov 03 11:14:11.909473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:11.909380    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181266b1-4afa-4379-8460-3a480576d67e" path="/var/lib/kubelet/pods/181266b1-4afa-4379-8460-3a480576d67e/volumes"
Nov 03 11:14:12.042168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:12.042082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:12.754593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:12.753136    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:13.372596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.370682    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:13.481660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.481620    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz" podUID="813cff89-38c5-46db-a92a-187b541220ee" containerName="payload-processing" containerID="cri-o://64003934817fa6f978d8a0d68e9e235950f30af5fc215f2fde6a2f0c48f9b434" gracePeriod=25
Nov 03 11:14:13.647700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.647664    5817 generic.go:356] "Generic (PLEG): container finished" podID="813cff89-38c5-46db-a92a-187b541220ee" containerID="64003934817fa6f978d8a0d68e9e235950f30af5fc215f2fde6a2f0c48f9b434" exitCode=0
Nov 03 11:14:13.647866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.647747    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz" event={"ID":"813cff89-38c5-46db-a92a-187b541220ee","Type":"ContainerDied","Data":"64003934817fa6f978d8a0d68e9e235950f30af5fc215f2fde6a2f0c48f9b434"}
Nov 03 11:14:13.797138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.797096    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:14:13.944810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.944780    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-kube-api-access-hfqp2\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-kube-api-access-hfqp2\") pod \"813cff89-38c5-46db-a92a-187b541220ee\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") "
Nov 03 11:14:13.945109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.945085    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-plugins-config-volume\") pod \"813cff89-38c5-46db-a92a-187b541220ee\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") "
Nov 03 11:14:13.945215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.945199    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-provider-credentials\") pod \"813cff89-38c5-46db-a92a-187b541220ee\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") "
Nov 03 11:14:13.945277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.945256    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-routing-overlay-volume\") pod \"813cff89-38c5-46db-a92a-187b541220ee\" (UID: \"813cff89-38c5-46db-a92a-187b541220ee\") "
Nov 03 11:14:13.945697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.945660    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-plugins-config-volume" pod "813cff89-38c5-46db-a92a-187b541220ee" (UID: "813cff89-38c5-46db-a92a-187b541220ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:13.945877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.945853    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-routing-overlay-volume" pod "813cff89-38c5-46db-a92a-187b541220ee" (UID: "813cff89-38c5-46db-a92a-187b541220ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:13.946034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.946016    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:13.946094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.946042    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/813cff89-38c5-46db-a92a-187b541220ee-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:13.947382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.947354    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-kube-api-access-hfqp2" pod "813cff89-38c5-46db-a92a-187b541220ee" (UID: "813cff89-38c5-46db-a92a-187b541220ee"). InnerVolumeSpecName "kube-api-access-hfqp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:13.947627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:13.947566    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-provider-credentials" pod "813cff89-38c5-46db-a92a-187b541220ee" (UID: "813cff89-38c5-46db-a92a-187b541220ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:14.049645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.048429    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:14.049645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.048468    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqp2\" (UniqueName: \"kubernetes.io/projected/813cff89-38c5-46db-a92a-187b541220ee-kube-api-access-hfqp2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:14.069602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.065193    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:14.556503 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.556463    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd" podUID="b3215f33-c070-49bc-ad4b-141c2b11e67d" containerName="payload-processing" containerID="cri-o://dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2" gracePeriod=25
Nov 03 11:14:14.683276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.683235    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"
Nov 03 11:14:14.683276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.683261    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz" event={"ID":"813cff89-38c5-46db-a92a-187b541220ee","Type":"ContainerDied","Data":"1bb499ee9a8c2550d1e44e0028e8c84cd52b2ee6e9547743879cc868424efce0"}
Nov 03 11:14:14.683513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.683323    5817 scope.go:122] "RemoveContainer" containerID="64003934817fa6f978d8a0d68e9e235950f30af5fc215f2fde6a2f0c48f9b434"
Nov 03 11:14:14.719106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.719078    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"]
Nov 03 11:14:14.725645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.725601    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-6456nz7rz"]
Nov 03 11:14:14.761522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:14.761279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:15.077222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.077180    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:15.210456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.210424    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:15.281859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.281791    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:14:15.441217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.425234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:15.449372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.446772    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-provider-credentials\") pod \"b3215f33-c070-49bc-ad4b-141c2b11e67d\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") "
Nov 03 11:14:15.449372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.446866    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-routing-overlay-volume\") pod \"b3215f33-c070-49bc-ad4b-141c2b11e67d\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") "
Nov 03 11:14:15.449372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.446965    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-kube-api-access-4d667\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-kube-api-access-4d667\") pod \"b3215f33-c070-49bc-ad4b-141c2b11e67d\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") "
Nov 03 11:14:15.449372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.447265    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-plugins-config-volume\") pod \"b3215f33-c070-49bc-ad4b-141c2b11e67d\" (UID: \"b3215f33-c070-49bc-ad4b-141c2b11e67d\") "
Nov 03 11:14:15.449372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.448415    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-routing-overlay-volume" pod "b3215f33-c070-49bc-ad4b-141c2b11e67d" (UID: "b3215f33-c070-49bc-ad4b-141c2b11e67d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:15.449372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.449274    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-plugins-config-volume" pod "b3215f33-c070-49bc-ad4b-141c2b11e67d" (UID: "b3215f33-c070-49bc-ad4b-141c2b11e67d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:15.451772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.451709    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:15.451772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.451741    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3215f33-c070-49bc-ad4b-141c2b11e67d-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:15.454080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.454038    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-provider-credentials" pod "b3215f33-c070-49bc-ad4b-141c2b11e67d" (UID: "b3215f33-c070-49bc-ad4b-141c2b11e67d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:15.454258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.454230    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-kube-api-access-4d667" pod "b3215f33-c070-49bc-ad4b-141c2b11e67d" (UID: "b3215f33-c070-49bc-ad4b-141c2b11e67d"). InnerVolumeSpecName "kube-api-access-4d667". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:15.553480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.553408    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d667\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-kube-api-access-4d667\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:15.553480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.553438    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3215f33-c070-49bc-ad4b-141c2b11e67d-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:15.567254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.567227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:15.696156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.696123    5817 generic.go:356] "Generic (PLEG): container finished" podID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerID="400e3df66e3806d2a15844a63c87f648c6fca1d31101706b6983758bcbd36c62" exitCode=0
Nov 03 11:14:15.696350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.696197    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" event={"ID":"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64","Type":"ContainerDied","Data":"400e3df66e3806d2a15844a63c87f648c6fca1d31101706b6983758bcbd36c62"}
Nov 03 11:14:15.696819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.696795    5817 scope.go:122] "RemoveContainer" containerID="400e3df66e3806d2a15844a63c87f648c6fca1d31101706b6983758bcbd36c62"
Nov 03 11:14:15.698238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.698217    5817 generic.go:356] "Generic (PLEG): container finished" podID="b3215f33-c070-49bc-ad4b-141c2b11e67d" containerID="dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2" exitCode=0
Nov 03 11:14:15.698350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.698268    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd" event={"ID":"b3215f33-c070-49bc-ad4b-141c2b11e67d","Type":"ContainerDied","Data":"dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2"}
Nov 03 11:14:15.698350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.698307    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd" event={"ID":"b3215f33-c070-49bc-ad4b-141c2b11e67d","Type":"ContainerDied","Data":"33f8f1e7c444d99c6fe3019f5a5b65db6f4d79b087972ef2d748690885da024e"}
Nov 03 11:14:15.698350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.698334    5817 scope.go:122] "RemoveContainer" containerID="dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2"
Nov 03 11:14:15.698522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.698460    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"
Nov 03 11:14:15.699361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.699326    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:15.699502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.699416    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:15.699502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.699465    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:15.723051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.723034    5817 scope.go:122] "RemoveContainer" containerID="dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2"
Nov 03 11:14:15.723423 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:15.723397    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2\": container with ID starting with dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2 not found: ID does not exist" containerID="dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2"
Nov 03 11:14:15.723520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.723434    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2"} err="failed to get container status \"dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2\": rpc error: code = NotFound desc = could not find container \"dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2\": container with ID starting with dac668e2ec9fb89afefdc23456f9dea8b557999a57f710fda071189aef2aa8a2 not found: ID does not exist"
Nov 03 11:14:15.763518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.763495    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"]
Nov 03 11:14:15.796626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.785408    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-776fxwpqd"]
Nov 03 11:14:15.917216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.917184    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813cff89-38c5-46db-a92a-187b541220ee" path="/var/lib/kubelet/pods/813cff89-38c5-46db-a92a-187b541220ee/volumes"
Nov 03 11:14:15.918819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.917751    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3215f33-c070-49bc-ad4b-141c2b11e67d" path="/var/lib/kubelet/pods/b3215f33-c070-49bc-ad4b-141c2b11e67d/volumes"
Nov 03 11:14:15.990680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.990641    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"]
Nov 03 11:14:15.992565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.992538    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2cfd3202-6582-471c-be21-211b5ace4aee" containerName="revoke-keys"
Nov 03 11:14:15.992755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.992741    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cfd3202-6582-471c-be21-211b5ace4aee" containerName="revoke-keys"
Nov 03 11:14:15.992899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.992887    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b3215f33-c070-49bc-ad4b-141c2b11e67d" containerName="payload-processing"
Nov 03 11:14:15.992989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.992978    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3215f33-c070-49bc-ad4b-141c2b11e67d" containerName="payload-processing"
Nov 03 11:14:15.993138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.993128    5817 cpu_manager.go:414] "containerMap: removing container" podUID="813cff89-38c5-46db-a92a-187b541220ee" containerName="payload-processing"
Nov 03 11:14:15.993214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.993205    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813cff89-38c5-46db-a92a-187b541220ee" containerName="payload-processing"
Nov 03 11:14:15.993526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.993512    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="813cff89-38c5-46db-a92a-187b541220ee" containerName="payload-processing"
Nov 03 11:14:15.993709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.993697    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cfd3202-6582-471c-be21-211b5ace4aee" containerName="revoke-keys"
Nov 03 11:14:15.993803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:15.993794    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3215f33-c070-49bc-ad4b-141c2b11e67d" containerName="payload-processing"
Nov 03 11:14:16.000774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.000748    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"]
Nov 03 11:14:16.000889 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.000866    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.013414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.013384    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a68103-openshift-default-dockercfg-t5bkr\""
Nov 03 11:14:16.041661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.041633    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"]
Nov 03 11:14:16.050627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.050507    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f96033-mhdq8"]
Nov 03 11:14:16.052301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.052277    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:16.168954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.168857    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-podinfo\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.169640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.169610    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-token\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.169846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.169828    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-certs\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.200661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.170024    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.201019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.200986    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-data\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.201235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.201217    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-envoy\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.201358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.201344    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9fce276-e6fe-4835-b146-c9a4642598e5-istiod-ca-cert\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.201531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.201516    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-credential-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.201891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.201865    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47t9x\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-kube-api-access-47t9x\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.217601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.216630    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:16.306803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.306756    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-credential-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.307651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.307623    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47t9x\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-kube-api-access-47t9x\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.308247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.308226    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-podinfo\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.308863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.308844    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-token\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.309308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.309289    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-certs\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.309996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.309976    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.310477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.310418    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-data\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.312060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.312042    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-envoy\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.312550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.312531    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9fce276-e6fe-4835-b146-c9a4642598e5-istiod-ca-cert\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.328189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.311915    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-data\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.328391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.307394    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-credential-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.328496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.310341    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.328641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.313526    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9fce276-e6fe-4835-b146-c9a4642598e5-istiod-ca-cert\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.328753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.309847    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-certs\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.331602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.329837    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-envoy\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.337063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.336215    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47t9x\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-kube-api-access-47t9x\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.337660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.337606    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-token\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.361593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.356099    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-podinfo\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.623411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.623370    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:16.626007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.625982    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:16.826081 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.825811    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" event={"ID":"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64","Type":"ContainerStarted","Data":"e463be0043bf19cbb832d2c21363cce5b7a11ecfc18d538d0d3064b31ae3777d"}
Nov 03 11:14:16.976200 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:14:16.976166    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fce276_e6fe_4835_b146_c9a4642598e5.slice/crio-947ac77e9f3e76ea2779d62843ee07dc7ccd7bbfe682af19a6a37e5532e46189 WatchSource:0}: Error finding container 947ac77e9f3e76ea2779d62843ee07dc7ccd7bbfe682af19a6a37e5532e46189: Status 404 returned error can't find the container with id 947ac77e9f3e76ea2779d62843ee07dc7ccd7bbfe682af19a6a37e5532e46189
Nov 03 11:14:16.978371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.978344    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"]
Nov 03 11:14:16.979389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.979359    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:16.979474 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.979424    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:16.979474 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:16.979454    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:17.276667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:17.276581    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:14:17.277042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:17.276758    5817 patch_prober.go:29] interesting pod/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 11:14:17.277042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:17.276794    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 11:14:17.461599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:17.461437    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"]
Nov 03 11:14:17.472105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:17.472041    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:17.718527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:17.718492    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:18.264613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.250956    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podStartSLOduration=3.250932361 podStartE2EDuration="3.250932361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:17.981544129 +0000 UTC m=+2688.187126224" watchObservedRunningTime="2026-11-03 11:14:18.250932361 +0000 UTC m=+2688.456514429"
Nov 03 11:14:18.378046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.376404    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d06c89-d29c-424f-a971-6fddacf7ea29" path="/var/lib/kubelet/pods/b0d06c89-d29c-424f-a971-6fddacf7ea29/volumes"
Nov 03 11:14:18.379320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.379296    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" event={"ID":"a9fce276-e6fe-4835-b146-c9a4642598e5","Type":"ContainerStarted","Data":"a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867"}
Nov 03 11:14:18.379458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.379444    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" event={"ID":"a9fce276-e6fe-4835-b146-c9a4642598e5","Type":"ContainerStarted","Data":"947ac77e9f3e76ea2779d62843ee07dc7ccd7bbfe682af19a6a37e5532e46189"}
Nov 03 11:14:18.382339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.382313    5817 patch_prober.go:29] interesting pod/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 11:14:18.382559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.382537    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 11:14:18.385753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.385724    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:18.465191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.465153    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"]
Nov 03 11:14:18.471089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.471059    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.471631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.470979    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"]
Nov 03 11:14:18.474888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.474864    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:14:18.474983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.474908    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:14:18.579491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.579022    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.579491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.579099    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.579491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.579288    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnd8\" (UniqueName: \"kubernetes.io/projected/724ad1c9-86b8-432c-a4d6-f61ea421a45d-kube-api-access-tqnd8\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.629000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.628967    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:18.635285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.635234    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:18.680988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.680954    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnd8\" (UniqueName: \"kubernetes.io/projected/724ad1c9-86b8-432c-a4d6-f61ea421a45d-kube-api-access-tqnd8\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.681210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.681194    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.681290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.681268    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.681985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.681963    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.683873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.683853    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.715592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.714834    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnd8\" (UniqueName: \"kubernetes.io/projected/724ad1c9-86b8-432c-a4d6-f61ea421a45d-kube-api-access-tqnd8\") pod \"maas-api-revoke-sub-03aa334b-jfg67\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.741698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.741479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:18.787632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.787549    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:18.938437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.938404    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:18.939557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:18.939535    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:14:19.035734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.035708    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"]
Nov 03 11:14:19.043304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.043275    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a30f38d-4bzk4"]
Nov 03 11:14:19.186243 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:14:19.186216    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724ad1c9_86b8_432c_a4d6_f61ea421a45d.slice/crio-496b9199094b20e6a3afb54fc237d14c13256e53448c08045a4cc135f83efd82 WatchSource:0}: Error finding container 496b9199094b20e6a3afb54fc237d14c13256e53448c08045a4cc135f83efd82: Status 404 returned error can't find the container with id 496b9199094b20e6a3afb54fc237d14c13256e53448c08045a4cc135f83efd82
Nov 03 11:14:19.193737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.187906    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"]
Nov 03 11:14:19.277976 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.277948    5817 patch_prober.go:29] interesting pod/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 11:14:19.278084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.278007    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 11:14:19.579673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.579635    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:19.760923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.760894    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:19.934746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.934714    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d5df9d-eea5-475d-ac9a-2e046d2ad737" path="/var/lib/kubelet/pods/d7d5df9d-eea5-475d-ac9a-2e046d2ad737/volumes"
Nov 03 11:14:19.950473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.950435    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67" event={"ID":"724ad1c9-86b8-432c-a4d6-f61ea421a45d","Type":"ContainerStarted","Data":"9980ccd714c08f3af5875965ee1206d7058bb30b953ea0e57263bbf6ef83a1a7"}
Nov 03 11:14:19.950662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.950487    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67" event={"ID":"724ad1c9-86b8-432c-a4d6-f61ea421a45d","Type":"ContainerStarted","Data":"496b9199094b20e6a3afb54fc237d14c13256e53448c08045a4cc135f83efd82"}
Nov 03 11:14:19.969508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:19.969461    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67" podStartSLOduration=1.969448806 podStartE2EDuration="1.969448806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:19.964809253 +0000 UTC m=+2690.170391304" watchObservedRunningTime="2026-11-03 11:14:19.969448806 +0000 UTC m=+2690.175030857"
Nov 03 11:14:20.276633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:20.276600    5817 patch_prober.go:29] interesting pod/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 11:14:20.276811 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:20.276653    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 11:14:20.328224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:20.328177    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:20.495642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:20.490116    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:20.650649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:20.642023    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:14:20.965425 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:20.965072    5817 generic.go:356] "Generic (PLEG): container finished" podID="724ad1c9-86b8-432c-a4d6-f61ea421a45d" containerID="9980ccd714c08f3af5875965ee1206d7058bb30b953ea0e57263bbf6ef83a1a7" exitCode=0
Nov 03 11:14:20.965425 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:20.965154    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67" event={"ID":"724ad1c9-86b8-432c-a4d6-f61ea421a45d","Type":"ContainerDied","Data":"9980ccd714c08f3af5875965ee1206d7058bb30b953ea0e57263bbf6ef83a1a7"}
Nov 03 11:14:21.276379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.276301    5817 patch_prober.go:29] interesting pod/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 11:14:21.276379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.276352    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 11:14:21.455989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.455810    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:21.616418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.615909    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:21.617943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.617917    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:14:21.624857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.624830    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"]
Nov 03 11:14:21.629543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.629499    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:21.646617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.645650    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"]
Nov 03 11:14:21.709610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.688646    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a68103-dockercfg-t696b\""
Nov 03 11:14:21.709610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.688743    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a68103\""
Nov 03 11:14:21.758650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.756631    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfrc\" (UniqueName: \"kubernetes.io/projected/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-kube-api-access-dvfrc\") pod \"payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs\" (UID: \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:21.758650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.756993    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs\" (UID: \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:21.893652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.884445    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs\" (UID: \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:21.945600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:21.941719    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs\" (UID: \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:22.090490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.090456    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfrc\" (UniqueName: \"kubernetes.io/projected/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-kube-api-access-dvfrc\") pod \"payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs\" (UID: \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:22.180640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.169310    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfrc\" (UniqueName: \"kubernetes.io/projected/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-kube-api-access-dvfrc\") pod \"payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs\" (UID: \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:22.222379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.222350    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"]
Nov 03 11:14:22.278182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.277155    5817 patch_prober.go:29] interesting pod/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 11:14:22.278182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.277218    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 11:14:22.419424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.414898    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:22.493991 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.474076    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v" podUID="6f90d72e-5f04-414d-95a2-0560d76f5c16" containerName="payload-processing" containerID="cri-o://e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d" gracePeriod=25
Nov 03 11:14:22.518611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.503586    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:22.532168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.529911    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:22.584388 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:22.584344    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:14:22.603593 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:14:22.602767    5817 logging.go:55] [core] [Channel #10180 SubChannel #10181]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.27:50052", ServerName: "10.133.1.27:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.27:50052: connect: connection refused"
Nov 03 11:14:22.666277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.666235    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-xks-ca\") pod \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") "
Nov 03 11:14:22.666403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.666356    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-openshift-ca\") pod \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") "
Nov 03 11:14:22.666484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.666411    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724ad1c9-86b8-432c-a4d6-f61ea421a45d-kube-api-access-tqnd8\" (UniqueName: \"kubernetes.io/projected/724ad1c9-86b8-432c-a4d6-f61ea421a45d-kube-api-access-tqnd8\") pod \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\" (UID: \"724ad1c9-86b8-432c-a4d6-f61ea421a45d\") "
Nov 03 11:14:22.666992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.666955    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-openshift-ca" pod "724ad1c9-86b8-432c-a4d6-f61ea421a45d" (UID: "724ad1c9-86b8-432c-a4d6-f61ea421a45d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:22.667554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.667527    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:22.669039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.669014    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724ad1c9-86b8-432c-a4d6-f61ea421a45d-kube-api-access-tqnd8" pod "724ad1c9-86b8-432c-a4d6-f61ea421a45d" (UID: "724ad1c9-86b8-432c-a4d6-f61ea421a45d"). InnerVolumeSpecName "kube-api-access-tqnd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:22.669414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.669383    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-xks-ca" pod "724ad1c9-86b8-432c-a4d6-f61ea421a45d" (UID: "724ad1c9-86b8-432c-a4d6-f61ea421a45d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:14:22.773693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.769647    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724ad1c9-86b8-432c-a4d6-f61ea421a45d-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:22.773693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.769678    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqnd8\" (UniqueName: \"kubernetes.io/projected/724ad1c9-86b8-432c-a4d6-f61ea421a45d-kube-api-access-tqnd8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:22.920134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.920098    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"]
Nov 03 11:14:22.920780 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:14:22.920722    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c3035c_e422_45aa_8a7e_100b3e46d1ab.slice/crio-24be6628127109abc0b703f4670cf7d5b8a91a7b954e25fbea489645a93564a0 WatchSource:0}: Error finding container 24be6628127109abc0b703f4670cf7d5b8a91a7b954e25fbea489645a93564a0: Status 404 returned error can't find the container with id 24be6628127109abc0b703f4670cf7d5b8a91a7b954e25fbea489645a93564a0
Nov 03 11:14:22.926104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:22.926083    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:14:23.074442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.074409    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-kube-api-access-b92w4\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-kube-api-access-b92w4\") pod \"6f90d72e-5f04-414d-95a2-0560d76f5c16\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") "
Nov 03 11:14:23.074650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.074474    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-plugins-config-volume\") pod \"6f90d72e-5f04-414d-95a2-0560d76f5c16\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") "
Nov 03 11:14:23.074650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.074563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-routing-overlay-volume\") pod \"6f90d72e-5f04-414d-95a2-0560d76f5c16\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") "
Nov 03 11:14:23.074794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.074726    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-provider-credentials\") pod \"6f90d72e-5f04-414d-95a2-0560d76f5c16\" (UID: \"6f90d72e-5f04-414d-95a2-0560d76f5c16\") "
Nov 03 11:14:23.076124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.076092    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-plugins-config-volume" pod "6f90d72e-5f04-414d-95a2-0560d76f5c16" (UID: "6f90d72e-5f04-414d-95a2-0560d76f5c16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:23.076360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.076343    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-routing-overlay-volume" pod "6f90d72e-5f04-414d-95a2-0560d76f5c16" (UID: "6f90d72e-5f04-414d-95a2-0560d76f5c16"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:23.077933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.077896    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-kube-api-access-b92w4" pod "6f90d72e-5f04-414d-95a2-0560d76f5c16" (UID: "6f90d72e-5f04-414d-95a2-0560d76f5c16"). InnerVolumeSpecName "kube-api-access-b92w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:23.079151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.079130    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-provider-credentials" pod "6f90d72e-5f04-414d-95a2-0560d76f5c16" (UID: "6f90d72e-5f04-414d-95a2-0560d76f5c16"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:23.177382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.177349    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:23.177382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.177380    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:23.177638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.177392    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b92w4\" (UniqueName: \"kubernetes.io/projected/6f90d72e-5f04-414d-95a2-0560d76f5c16-kube-api-access-b92w4\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:23.177638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.177401    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f90d72e-5f04-414d-95a2-0560d76f5c16-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:23.234861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.234825    5817 generic.go:356] "Generic (PLEG): container finished" podID="6f90d72e-5f04-414d-95a2-0560d76f5c16" containerID="e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d" exitCode=0
Nov 03 11:14:23.235188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.234864    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"
Nov 03 11:14:23.235355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.234888    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v" event={"ID":"6f90d72e-5f04-414d-95a2-0560d76f5c16","Type":"ContainerDied","Data":"e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d"}
Nov 03 11:14:23.235355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.235289    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v" event={"ID":"6f90d72e-5f04-414d-95a2-0560d76f5c16","Type":"ContainerDied","Data":"05f153d3b0576f26d5db398776200cc2afb8822b2b5cc20bbc01f0448aac99ca"}
Nov 03 11:14:23.235355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.235315    5817 scope.go:122] "RemoveContainer" containerID="e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d"
Nov 03 11:14:23.238385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.238361    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs" event={"ID":"c6c3035c-e422-45aa-8a7e-100b3e46d1ab","Type":"ContainerStarted","Data":"dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10"}
Nov 03 11:14:23.238497 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.238404    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs" event={"ID":"c6c3035c-e422-45aa-8a7e-100b3e46d1ab","Type":"ContainerStarted","Data":"24be6628127109abc0b703f4670cf7d5b8a91a7b954e25fbea489645a93564a0"}
Nov 03 11:14:23.239204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.239183    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:23.240616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.240557    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67" event={"ID":"724ad1c9-86b8-432c-a4d6-f61ea421a45d","Type":"ContainerDied","Data":"496b9199094b20e6a3afb54fc237d14c13256e53448c08045a4cc135f83efd82"}
Nov 03 11:14:23.240720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.240637    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="496b9199094b20e6a3afb54fc237d14c13256e53448c08045a4cc135f83efd82"
Nov 03 11:14:23.240720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.240682    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"
Nov 03 11:14:23.255616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.255599    5817 scope.go:122] "RemoveContainer" containerID="e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d"
Nov 03 11:14:23.255918 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:23.255897    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d\": container with ID starting with e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d not found: ID does not exist" containerID="e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d"
Nov 03 11:14:23.255974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.255927    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d"} err="failed to get container status \"e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d\": rpc error: code = NotFound desc = could not find container \"e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d\": container with ID starting with e6a9d10baed327ba7777f1850fc221914d650a0ab82c8d1af1cf1d6614f8941d not found: ID does not exist"
Nov 03 11:14:23.348649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.283439    5817 patch_prober.go:29] interesting pod/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 11:14:23.348649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.283524    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 11:14:23.348649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.287453    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs" podStartSLOduration=2.287434139 podStartE2EDuration="2.287434139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:23.258292843 +0000 UTC m=+2693.463874891" watchObservedRunningTime="2026-11-03 11:14:23.287434139 +0000 UTC m=+2693.493016191"
Nov 03 11:14:23.348649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.316380    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:23.425599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.424077    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"]
Nov 03 11:14:23.535057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.534133    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v"]
Nov 03 11:14:23.594199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.594134    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-84c8zc92v" podUID="6f90d72e-5f04-414d-95a2-0560d76f5c16" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.27:50052\" within 1s: context deadline exceeded"
Nov 03 11:14:23.909862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.909827    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f90d72e-5f04-414d-95a2-0560d76f5c16" path="/var/lib/kubelet/pods/6f90d72e-5f04-414d-95a2-0560d76f5c16/volumes"
Nov 03 11:14:23.994649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:23.991097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:24.281881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:24.281851    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:14:24.282348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:24.282316    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:14:24.283155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:24.283135    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:14:24.593308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:24.593224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:25.327821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:25.327374    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:25.959607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:25.953025    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:26.026358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:26.026323    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"]
Nov 03 11:14:26.031586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:26.031545    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb555f4c-rq896"]
Nov 03 11:14:26.643609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:26.642642    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:27.223648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.223607    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj" podUID="9d76f4dd-9618-4ead-8cd2-b999526c1454" containerName="payload-processing" containerID="cri-o://dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b" gracePeriod=25
Nov 03 11:14:27.309792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.305482    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:27.362485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.362174    5817 generic.go:356] "Generic (PLEG): container finished" podID="9d76f4dd-9618-4ead-8cd2-b999526c1454" containerID="dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b" exitCode=0
Nov 03 11:14:27.362485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.362228    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj" event={"ID":"9d76f4dd-9618-4ead-8cd2-b999526c1454","Type":"ContainerDied","Data":"dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b"}
Nov 03 11:14:27.370892 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:27.369796    5817 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d76f4dd_9618_4ead_8cd2_b999526c1454.slice/crio-dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d76f4dd_9618_4ead_8cd2_b999526c1454.slice/crio-conmon-dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:27.370892 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:27.370762    5817 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d76f4dd_9618_4ead_8cd2_b999526c1454.slice/crio-dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d76f4dd_9618_4ead_8cd2_b999526c1454.slice/crio-conmon-dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:27.541314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.541289    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:14:27.639974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.639933    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-kube-api-access-vfq52\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-kube-api-access-vfq52\") pod \"9d76f4dd-9618-4ead-8cd2-b999526c1454\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") "
Nov 03 11:14:27.640213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.640197    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-plugins-config-volume\") pod \"9d76f4dd-9618-4ead-8cd2-b999526c1454\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") "
Nov 03 11:14:27.640264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.640252    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-routing-overlay-volume\") pod \"9d76f4dd-9618-4ead-8cd2-b999526c1454\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") "
Nov 03 11:14:27.640589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.640556    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-provider-credentials\") pod \"9d76f4dd-9618-4ead-8cd2-b999526c1454\" (UID: \"9d76f4dd-9618-4ead-8cd2-b999526c1454\") "
Nov 03 11:14:27.640706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.640640    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-plugins-config-volume" pod "9d76f4dd-9618-4ead-8cd2-b999526c1454" (UID: "9d76f4dd-9618-4ead-8cd2-b999526c1454"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:27.640706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.640673    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-routing-overlay-volume" pod "9d76f4dd-9618-4ead-8cd2-b999526c1454" (UID: "9d76f4dd-9618-4ead-8cd2-b999526c1454"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:27.641108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.641086    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:27.641108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.641105    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d76f4dd-9618-4ead-8cd2-b999526c1454-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:27.642218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.642192    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-kube-api-access-vfq52" pod "9d76f4dd-9618-4ead-8cd2-b999526c1454" (UID: "9d76f4dd-9618-4ead-8cd2-b999526c1454"). InnerVolumeSpecName "kube-api-access-vfq52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:27.642788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.642765    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-provider-credentials" pod "9d76f4dd-9618-4ead-8cd2-b999526c1454" (UID: "9d76f4dd-9618-4ead-8cd2-b999526c1454"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:27.742758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.742725    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:27.742758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.742751    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfq52\" (UniqueName: \"kubernetes.io/projected/9d76f4dd-9618-4ead-8cd2-b999526c1454-kube-api-access-vfq52\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:27.912792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.911755    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:27.956357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:27.956316    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d7027a-7718-4395-abbe-de6aba1058c1" path="/var/lib/kubelet/pods/c2d7027a-7718-4395-abbe-de6aba1058c1/volumes"
Nov 03 11:14:28.374390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:28.374363    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"
Nov 03 11:14:28.374390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:28.374377    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj" event={"ID":"9d76f4dd-9618-4ead-8cd2-b999526c1454","Type":"ContainerDied","Data":"8ade60c1a422914833f60bd84bfbdeee855f59258265d6fba54302b24cc70095"}
Nov 03 11:14:28.374703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:28.374448    5817 scope.go:122] "RemoveContainer" containerID="dd67992d0f41cf30a72e5ea51e3f837c18818557fc7210a87d82f367a838297b"
Nov 03 11:14:28.397199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:28.397170    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"]
Nov 03 11:14:28.402732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:28.402706    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfjhkzj"]
Nov 03 11:14:28.587822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:28.586556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:29.134701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:29.134666    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:29.296600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:29.296191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:29.687869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:29.687828    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:29.910228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:29.910191    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d76f4dd-9618-4ead-8cd2-b999526c1454" path="/var/lib/kubelet/pods/9d76f4dd-9618-4ead-8cd2-b999526c1454/volumes"
Nov 03 11:14:30.065685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:30.065589    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:30.764602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:30.762774    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:31.371513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:31.371476    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:32.022258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:32.022219    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:32.635412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:32.635378    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:33.255144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:33.255108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:33.989890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:33.989854    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:34.617381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:34.617287    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:35.286202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:35.286165    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:35.331492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:35.331470    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:14:35.859415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:35.859367    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:36.009661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:36.009620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:36.156386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:36.156302    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:36.302907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:36.302872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:36.623322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:36.623281    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:36.760282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:36.760242    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:37.235433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:37.235396    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:37.384926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:37.384836    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:37.792602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:37.790107    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:37.976456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:37.976422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:38.631476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:38.631442    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:38.859597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:38.828908    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:14:38.859597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:38.850522    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:14:39.610865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:39.610541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:39.759156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:39.759122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:40.307139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:40.299804    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:40.471052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:40.471017    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:40.646304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:40.646212    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:41.029196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:41.029162    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"]
Nov 03 11:14:41.040234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:41.040199    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4273ab-zs8t8"]
Nov 03 11:14:41.370422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:41.370329    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:41.909274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:41.909236    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6b8434-ffdb-4383-9d8e-b7a9811c6449" path="/var/lib/kubelet/pods/fd6b8434-ffdb-4383-9d8e-b7a9811c6449/volumes"
Nov 03 11:14:41.971608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:41.968710    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:42.032660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.032631    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"]
Nov 03 11:14:42.038090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.038064    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99ec346-5tx5g"]
Nov 03 11:14:42.552971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.552939    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"]
Nov 03 11:14:42.554361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554334    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6f90d72e-5f04-414d-95a2-0560d76f5c16" containerName="payload-processing"
Nov 03 11:14:42.554478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554363    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f90d72e-5f04-414d-95a2-0560d76f5c16" containerName="payload-processing"
Nov 03 11:14:42.554478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554402    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9d76f4dd-9618-4ead-8cd2-b999526c1454" containerName="payload-processing"
Nov 03 11:14:42.554478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554410    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d76f4dd-9618-4ead-8cd2-b999526c1454" containerName="payload-processing"
Nov 03 11:14:42.554478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554424    5817 cpu_manager.go:414] "containerMap: removing container" podUID="724ad1c9-86b8-432c-a4d6-f61ea421a45d" containerName="revoke-keys"
Nov 03 11:14:42.554478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554431    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724ad1c9-86b8-432c-a4d6-f61ea421a45d" containerName="revoke-keys"
Nov 03 11:14:42.554802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554673    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f90d72e-5f04-414d-95a2-0560d76f5c16" containerName="payload-processing"
Nov 03 11:14:42.554802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554698    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d76f4dd-9618-4ead-8cd2-b999526c1454" containerName="payload-processing"
Nov 03 11:14:42.554802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.554711    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="724ad1c9-86b8-432c-a4d6-f61ea421a45d" containerName="revoke-keys"
Nov 03 11:14:42.561319 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.561300    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.567165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.566746    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a68103-serving-cert\""
Nov 03 11:14:42.568727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.568702    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"]
Nov 03 11:14:42.634674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.634643    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.634984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.634958    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab02689b-7c6e-4341-91b4-269a7c51e346-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.635033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.635020    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbdd\" (UniqueName: \"kubernetes.io/projected/ab02689b-7c6e-4341-91b4-269a7c51e346-kube-api-access-nxbdd\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.635224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.635202    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-metrics-certs\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.651589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.651543    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:42.742126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.742075    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab02689b-7c6e-4341-91b4-269a7c51e346-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.742352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.742148    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbdd\" (UniqueName: \"kubernetes.io/projected/ab02689b-7c6e-4341-91b4-269a7c51e346-kube-api-access-nxbdd\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.742352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.742292    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-metrics-certs\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.742481 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.742394    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.742720 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:42.742697    5817 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-serving-cert: secret "maas-api-e2e-inf-a-a68103-serving-cert" not found
Nov 03 11:14:42.742853 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:42.742804    5817 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls podName:ab02689b-7c6e-4341-91b4-269a7c51e346 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:43.242784628 +0000 UTC m=+2713.448366679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls") pod "maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg" (UID: "ab02689b-7c6e-4341-91b4-269a7c51e346") : secret "maas-api-e2e-inf-a-a68103-serving-cert" not found
Nov 03 11:14:42.760664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.760545    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-metrics-certs\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.771637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.770199    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbdd\" (UniqueName: \"kubernetes.io/projected/ab02689b-7c6e-4341-91b4-269a7c51e346-kube-api-access-nxbdd\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.782813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.782781    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab02689b-7c6e-4341-91b4-269a7c51e346-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:42.783623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.783549    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:42.910759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:42.910669    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:43.258599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:43.258225    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:43.316605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:43.314233    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls\") pod \"maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:43.481289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:43.481253    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:43.619395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:43.614206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:43.776868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:43.776822    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:43.781605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:43.781559    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"]
Nov 03 11:14:43.783149 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:14:43.783125    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab02689b_7c6e_4341_91b4_269a7c51e346.slice/crio-07213194218b94a473a318015f47b2952c95daa94c3a8e2ef89599d54b15b222 WatchSource:0}: Error finding container 07213194218b94a473a318015f47b2952c95daa94c3a8e2ef89599d54b15b222: Status 404 returned error can't find the container with id 07213194218b94a473a318015f47b2952c95daa94c3a8e2ef89599d54b15b222
Nov 03 11:14:43.905544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:43.905461    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d" path="/var/lib/kubelet/pods/e6ac1e9c-ddda-4ddc-88f2-d02a114cdb9d/volumes"
Nov 03 11:14:44.340611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:44.333587    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:44.546430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:44.546347    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg" event={"ID":"ab02689b-7c6e-4341-91b4-269a7c51e346","Type":"ContainerStarted","Data":"7a04e4e153fc7d2c044ff407c01d8777cf1a9d1ff1a9ebc495dc4db9c0176c1a"}
Nov 03 11:14:44.546430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:44.546419    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg" event={"ID":"ab02689b-7c6e-4341-91b4-269a7c51e346","Type":"ContainerStarted","Data":"07213194218b94a473a318015f47b2952c95daa94c3a8e2ef89599d54b15b222"}
Nov 03 11:14:44.546648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:44.546455    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:44.569389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:44.569346    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg" podStartSLOduration=2.089019416 podStartE2EDuration="2.56933398s" totalImagesPullingTime="480.314564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:44.564408524 +0000 UTC m=+2714.769990576" watchObservedRunningTime="2026-11-03 11:14:44.56933398 +0000 UTC m=+2714.774916031"
Nov 03 11:14:45.122334 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:45.122294    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:45.198835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:45.198801    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:14:45.832627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:45.827131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:46.048339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:46.048309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:46.821843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:46.821750    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:47.002006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:47.001974    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:47.286919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:47.286887    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:47.778611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:47.778448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:48.237658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:48.185911    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:48.437268 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:48.437222    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:49.324267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:49.324235    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:50.238776 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:50.238737    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:50.561118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:50.561039    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:14:51.228597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:51.218262    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:51.848098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:51.848058    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:51.989598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:51.985936    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:52.421502 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:14:52.421463    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:14:52.814549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:52.814515    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:53.078181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.078086    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:53.130732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.130692    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"]
Nov 03 11:14:53.136358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.136328    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:53.149224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.148944    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"]
Nov 03 11:14:53.211637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.205102    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-055d82-dockercfg-4dgcg\""
Nov 03 11:14:53.211637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.205232    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-055d82\""
Nov 03 11:14:53.278596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.239759    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:53.278596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.253271    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd4vc\" (UniqueName: \"kubernetes.io/projected/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-kube-api-access-kd4vc\") pod \"payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj\" (UID: \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:53.278596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.253510    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj\" (UID: \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:53.387708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.387344    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd4vc\" (UniqueName: \"kubernetes.io/projected/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-kube-api-access-kd4vc\") pod \"payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj\" (UID: \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:53.478598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.422416    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj\" (UID: \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:53.519666 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.508210    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj\" (UID: \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:53.563868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.563780    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd4vc\" (UniqueName: \"kubernetes.io/projected/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-kube-api-access-kd4vc\") pod \"payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj\" (UID: \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:53.831743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:53.831707    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:54.066665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:54.066587    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:54.138382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:54.138295    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"]
Nov 03 11:14:54.144895 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:14:54.144867    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5d9e99_aa64_4a6d_a11d_1617ef5e643e.slice/crio-25a71fcb91f6ef9ed38fcd778a551c9c123e8ed1219af9161de3cc9baf10cbbb WatchSource:0}: Error finding container 25a71fcb91f6ef9ed38fcd778a551c9c123e8ed1219af9161de3cc9baf10cbbb: Status 404 returned error can't find the container with id 25a71fcb91f6ef9ed38fcd778a551c9c123e8ed1219af9161de3cc9baf10cbbb
Nov 03 11:14:54.766592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:54.766468    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:54.804595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:54.802650    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj" event={"ID":"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e","Type":"ContainerStarted","Data":"e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487"}
Nov 03 11:14:54.804595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:54.802749    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj" event={"ID":"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e","Type":"ContainerStarted","Data":"25a71fcb91f6ef9ed38fcd778a551c9c123e8ed1219af9161de3cc9baf10cbbb"}
Nov 03 11:14:54.804595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:54.804402    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:14:54.828852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:54.828803    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj" podStartSLOduration=1.828786099 podStartE2EDuration="1.828786099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:54.826253837 +0000 UTC m=+2725.031835908" watchObservedRunningTime="2026-11-03 11:14:54.828786099 +0000 UTC m=+2725.034368150"
Nov 03 11:14:55.392607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:55.382519    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:56.098240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:56.098151    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:56.676199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:56.676168    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:57.351556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:57.351521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:57.993029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:57.992994    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:58.232730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:58.232688    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:58.596623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:58.589108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:59.232909 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:59.232820    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:14:59.886758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:14:59.886717    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:00.138385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.138299    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"]
Nov 03 11:15:00.144886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.144862    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:00.145273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.145251    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"]
Nov 03 11:15:00.154419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.154389    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:00.157241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.157212    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"]
Nov 03 11:15:00.165133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.165109    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:15:00.211027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.199660    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"]
Nov 03 11:15:00.211027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.199709    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"]
Nov 03 11:15:00.211027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.199735    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"]
Nov 03 11:15:00.211027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.199828    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:00.211027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.206305    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386c604-04c2-4d21-bd0d-99b4bc965c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs\" (UID: \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:00.211027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.206344    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hppc\" (UniqueName: \"kubernetes.io/projected/8f21b3aa-337d-46be-ae7a-c5658f48b0af-kube-api-access-6hppc\") pod \"maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc\" (UID: \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:00.211027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.206367    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21b3aa-337d-46be-ae7a-c5658f48b0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc\" (UID: \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:00.211027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.206383    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tswvt\" (UniqueName: \"kubernetes.io/projected/a386c604-04c2-4d21-bd0d-99b4bc965c3e-kube-api-access-tswvt\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs\" (UID: \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:00.214427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.212158    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"]
Nov 03 11:15:00.220535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.219418    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"]
Nov 03 11:15:00.220715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.220543    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:00.248958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.248931    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:00.260313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.260267    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"]
Nov 03 11:15:00.311398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.311365    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr\" (UID: \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:00.311778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.311756    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnwx7\" (UniqueName: \"kubernetes.io/projected/24da8686-666d-4c92-81fa-5e6a07e4c5bb-kube-api-access-wnwx7\") pod \"maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss\" (UID: \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:00.311918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.311900    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfph\" (UniqueName: \"kubernetes.io/projected/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-kube-api-access-xqfph\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5\" (UID: \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:00.312086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.312070    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386c604-04c2-4d21-bd0d-99b4bc965c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs\" (UID: \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:00.312218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.312202    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5\" (UID: \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:00.312332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.312317    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hppc\" (UniqueName: \"kubernetes.io/projected/8f21b3aa-337d-46be-ae7a-c5658f48b0af-kube-api-access-6hppc\") pod \"maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc\" (UID: \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:00.312442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.312429    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21b3aa-337d-46be-ae7a-c5658f48b0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc\" (UID: \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:00.312537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.312523    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tswvt\" (UniqueName: \"kubernetes.io/projected/a386c604-04c2-4d21-bd0d-99b4bc965c3e-kube-api-access-tswvt\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs\" (UID: \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:00.319461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.313029    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"]
Nov 03 11:15:00.319461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.314208    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21b3aa-337d-46be-ae7a-c5658f48b0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc\" (UID: \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:00.319461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.314908    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386c604-04c2-4d21-bd0d-99b4bc965c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs\" (UID: \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:00.319461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.316651    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"]
Nov 03 11:15:00.319835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.319815    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2rl\" (UniqueName: \"kubernetes.io/projected/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-kube-api-access-cq2rl\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr\" (UID: \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:00.319993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.319965    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24da8686-666d-4c92-81fa-5e6a07e4c5bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss\" (UID: \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:00.325165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.325141    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:00.327455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.327430    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tswvt\" (UniqueName: \"kubernetes.io/projected/a386c604-04c2-4d21-bd0d-99b4bc965c3e-kube-api-access-tswvt\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs\" (UID: \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:00.329746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.329031    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"]
Nov 03 11:15:00.329746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.329295    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hppc\" (UniqueName: \"kubernetes.io/projected/8f21b3aa-337d-46be-ae7a-c5658f48b0af-kube-api-access-6hppc\") pod \"maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc\" (UID: \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:00.335942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.335918    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:00.342869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.342400    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"]
Nov 03 11:15:00.352600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.352559    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"]
Nov 03 11:15:00.352708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.352690    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:00.370411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.370370    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"]
Nov 03 11:15:00.382393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.382362    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"]
Nov 03 11:15:00.396534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.395989    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:00.398090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.398064    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"]
Nov 03 11:15:00.405159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.405137    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:00.413184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.413164    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"]
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.421800    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skg26\" (UniqueName: \"kubernetes.io/projected/6da14868-435e-4ed1-96a3-b295fa1ac1a1-kube-api-access-skg26\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk\" (UID: \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.421838    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp\" (UID: \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.421933    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq8t2\" (UniqueName: \"kubernetes.io/projected/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-kube-api-access-dq8t2\") pod \"maas-api-key-cleanup-29895075-tvldd\" (UID: \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.421971    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr\" (UID: \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422016    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnwx7\" (UniqueName: \"kubernetes.io/projected/24da8686-666d-4c92-81fa-5e6a07e4c5bb-kube-api-access-wnwx7\") pod \"maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss\" (UID: \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422036    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfph\" (UniqueName: \"kubernetes.io/projected/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-kube-api-access-xqfph\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5\" (UID: \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422060    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjjv\" (UniqueName: \"kubernetes.io/projected/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-kube-api-access-xnjjv\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp\" (UID: \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422109    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5\" (UID: \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422138    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14868-435e-4ed1-96a3-b295fa1ac1a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk\" (UID: \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422215    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2rl\" (UniqueName: \"kubernetes.io/projected/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-kube-api-access-cq2rl\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr\" (UID: \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422242    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24da8686-666d-4c92-81fa-5e6a07e4c5bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss\" (UID: \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422307    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895075-tvldd\" (UID: \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:00.422990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.422649    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"]
Nov 03 11:15:00.424073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.423566    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5\" (UID: \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:00.425677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.424264    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24da8686-666d-4c92-81fa-5e6a07e4c5bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss\" (UID: \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:00.425677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.424675    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr\" (UID: \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:00.430438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.430413    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"]
Nov 03 11:15:00.435031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.435006    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfph\" (UniqueName: \"kubernetes.io/projected/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-kube-api-access-xqfph\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5\" (UID: \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:00.436007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.435984    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnwx7\" (UniqueName: \"kubernetes.io/projected/24da8686-666d-4c92-81fa-5e6a07e4c5bb-kube-api-access-wnwx7\") pod \"maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss\" (UID: \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:00.436178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.436151    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2rl\" (UniqueName: \"kubernetes.io/projected/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-kube-api-access-cq2rl\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr\" (UID: \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:00.436949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.436927    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:00.437365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.437346    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"]
Nov 03 11:15:00.443755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.443737    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"]
Nov 03 11:15:00.443885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.443865    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:00.448471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.448454    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"]
Nov 03 11:15:00.448563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.448524    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:00.458200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.458178    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"]
Nov 03 11:15:00.462964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.462942    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"]
Nov 03 11:15:00.470141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.470121    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"]
Nov 03 11:15:00.470230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.470220    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:00.473651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.473631    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"]
Nov 03 11:15:00.478684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.478660    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"]
Nov 03 11:15:00.486193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.486171    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:00.489242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.489217    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"]
Nov 03 11:15:00.494998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.494963    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"]
Nov 03 11:15:00.526184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526157    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0566a2-4971-4901-a926-af80b11a0603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd\" (UID: \"2d0566a2-4971-4901-a926-af80b11a0603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:00.526308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526206    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895075-tvldd\" (UID: \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:00.526389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526364    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vn84\" (UniqueName: \"kubernetes.io/projected/fbc01256-4c3b-40ce-9b18-7b20596c7255-kube-api-access-2vn84\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t\" (UID: \"fbc01256-4c3b-40ce-9b18-7b20596c7255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:00.526455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526407    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a0333b-9ed5-4e5a-95cf-ea734272085b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c\" (UID: \"63a0333b-9ed5-4e5a-95cf-ea734272085b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:00.526516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526464    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqdr\" (UniqueName: \"kubernetes.io/projected/2d0566a2-4971-4901-a926-af80b11a0603-kube-api-access-gsqdr\") pod \"maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd\" (UID: \"2d0566a2-4971-4901-a926-af80b11a0603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:00.526516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526495    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-kube-api-access-gd5kn\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr\" (UID: \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:00.526649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526549    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc01256-4c3b-40ce-9b18-7b20596c7255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t\" (UID: \"fbc01256-4c3b-40ce-9b18-7b20596c7255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:00.526649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526606    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a126cd-c401-485f-90f9-eaeb57262e03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl\" (UID: \"d9a126cd-c401-485f-90f9-eaeb57262e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:00.526815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526691    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skg26\" (UniqueName: \"kubernetes.io/projected/6da14868-435e-4ed1-96a3-b295fa1ac1a1-kube-api-access-skg26\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk\" (UID: \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:00.526815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526738    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp\" (UID: \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:00.526815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526805    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv\" (UID: \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:00.526969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526847    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn96v\" (UniqueName: \"kubernetes.io/projected/63a0333b-9ed5-4e5a-95cf-ea734272085b-kube-api-access-vn96v\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c\" (UID: \"63a0333b-9ed5-4e5a-95cf-ea734272085b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:00.526969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526852    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895075-tvldd\" (UID: \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:00.526969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.526882    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq8t2\" (UniqueName: \"kubernetes.io/projected/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-kube-api-access-dq8t2\") pod \"maas-api-key-cleanup-29895075-tvldd\" (UID: \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:00.527526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.527503    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp\" (UID: \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:00.527526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.527505    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjjv\" (UniqueName: \"kubernetes.io/projected/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-kube-api-access-xnjjv\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp\" (UID: \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:00.527728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.527588    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr\" (UID: \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:00.527728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.527632    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14868-435e-4ed1-96a3-b295fa1ac1a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk\" (UID: \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:00.527728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.527667    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5kp\" (UniqueName: \"kubernetes.io/projected/d9a126cd-c401-485f-90f9-eaeb57262e03-kube-api-access-zf5kp\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl\" (UID: \"d9a126cd-c401-485f-90f9-eaeb57262e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:00.527728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.527714    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhzl\" (UniqueName: \"kubernetes.io/projected/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-kube-api-access-2qhzl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv\" (UID: \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:00.528302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.528282    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14868-435e-4ed1-96a3-b295fa1ac1a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk\" (UID: \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:00.538184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.538161    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjjv\" (UniqueName: \"kubernetes.io/projected/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-kube-api-access-xnjjv\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp\" (UID: \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:00.538408 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.538385    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skg26\" (UniqueName: \"kubernetes.io/projected/6da14868-435e-4ed1-96a3-b295fa1ac1a1-kube-api-access-skg26\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk\" (UID: \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:00.538520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.538502    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq8t2\" (UniqueName: \"kubernetes.io/projected/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-kube-api-access-dq8t2\") pod \"maas-api-key-cleanup-29895075-tvldd\" (UID: \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:00.573439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.573412    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:00.580737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.580714    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:00.587011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.586989    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:00.622224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.622056    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.629843    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0967a8b-43f1-496a-8a5c-03e435f50232-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc\" (UID: \"f0967a8b-43f1-496a-8a5c-03e435f50232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.629925    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0566a2-4971-4901-a926-af80b11a0603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd\" (UID: \"2d0566a2-4971-4901-a926-af80b11a0603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630041    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vn84\" (UniqueName: \"kubernetes.io/projected/fbc01256-4c3b-40ce-9b18-7b20596c7255-kube-api-access-2vn84\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t\" (UID: \"fbc01256-4c3b-40ce-9b18-7b20596c7255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630072    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a0333b-9ed5-4e5a-95cf-ea734272085b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c\" (UID: \"63a0333b-9ed5-4e5a-95cf-ea734272085b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630104    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9tlh\" (UniqueName: \"kubernetes.io/projected/f0967a8b-43f1-496a-8a5c-03e435f50232-kube-api-access-p9tlh\") pod \"maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc\" (UID: \"f0967a8b-43f1-496a-8a5c-03e435f50232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630151    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqdr\" (UniqueName: \"kubernetes.io/projected/2d0566a2-4971-4901-a926-af80b11a0603-kube-api-access-gsqdr\") pod \"maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd\" (UID: \"2d0566a2-4971-4901-a926-af80b11a0603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630180    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-kube-api-access-gd5kn\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr\" (UID: \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630230    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc01256-4c3b-40ce-9b18-7b20596c7255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t\" (UID: \"fbc01256-4c3b-40ce-9b18-7b20596c7255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630265    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a126cd-c401-485f-90f9-eaeb57262e03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl\" (UID: \"d9a126cd-c401-485f-90f9-eaeb57262e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630385    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv\" (UID: \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630422    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn96v\" (UniqueName: \"kubernetes.io/projected/63a0333b-9ed5-4e5a-95cf-ea734272085b-kube-api-access-vn96v\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c\" (UID: \"63a0333b-9ed5-4e5a-95cf-ea734272085b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630610    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr\" (UID: \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630661    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5kp\" (UniqueName: \"kubernetes.io/projected/d9a126cd-c401-485f-90f9-eaeb57262e03-kube-api-access-zf5kp\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl\" (UID: \"d9a126cd-c401-485f-90f9-eaeb57262e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.630705    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhzl\" (UniqueName: \"kubernetes.io/projected/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-kube-api-access-2qhzl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv\" (UID: \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.631839    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0566a2-4971-4901-a926-af80b11a0603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd\" (UID: \"2d0566a2-4971-4901-a926-af80b11a0603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:00.633605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.632782    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a0333b-9ed5-4e5a-95cf-ea734272085b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c\" (UID: \"63a0333b-9ed5-4e5a-95cf-ea734272085b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:00.634926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.633730    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:00.636038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.635763    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a126cd-c401-485f-90f9-eaeb57262e03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl\" (UID: \"d9a126cd-c401-485f-90f9-eaeb57262e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:00.636038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.635928    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc01256-4c3b-40ce-9b18-7b20596c7255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t\" (UID: \"fbc01256-4c3b-40ce-9b18-7b20596c7255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:00.636038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.635978    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv\" (UID: \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:00.636780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.636387    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr\" (UID: \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:00.640470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.640409    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vn84\" (UniqueName: \"kubernetes.io/projected/fbc01256-4c3b-40ce-9b18-7b20596c7255-kube-api-access-2vn84\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t\" (UID: \"fbc01256-4c3b-40ce-9b18-7b20596c7255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:00.643899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.643823    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn96v\" (UniqueName: \"kubernetes.io/projected/63a0333b-9ed5-4e5a-95cf-ea734272085b-kube-api-access-vn96v\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c\" (UID: \"63a0333b-9ed5-4e5a-95cf-ea734272085b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:00.644517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.644497    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5kp\" (UniqueName: \"kubernetes.io/projected/d9a126cd-c401-485f-90f9-eaeb57262e03-kube-api-access-zf5kp\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl\" (UID: \"d9a126cd-c401-485f-90f9-eaeb57262e03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:00.646341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.645543    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqdr\" (UniqueName: \"kubernetes.io/projected/2d0566a2-4971-4901-a926-af80b11a0603-kube-api-access-gsqdr\") pod \"maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd\" (UID: \"2d0566a2-4971-4901-a926-af80b11a0603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:00.646341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.646279    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhzl\" (UniqueName: \"kubernetes.io/projected/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-kube-api-access-2qhzl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv\" (UID: \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:00.650784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.650742    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-kube-api-access-gd5kn\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr\" (UID: \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:00.724729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.694464    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:00.724729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.696799    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:00.724729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.704550    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:00.750518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.746673    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:00.750518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.746853    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:00.750518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.747556    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0967a8b-43f1-496a-8a5c-03e435f50232-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc\" (UID: \"f0967a8b-43f1-496a-8a5c-03e435f50232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:00.750518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.747707    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9tlh\" (UniqueName: \"kubernetes.io/projected/f0967a8b-43f1-496a-8a5c-03e435f50232-kube-api-access-p9tlh\") pod \"maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc\" (UID: \"f0967a8b-43f1-496a-8a5c-03e435f50232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:00.750518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.748985    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0967a8b-43f1-496a-8a5c-03e435f50232-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc\" (UID: \"f0967a8b-43f1-496a-8a5c-03e435f50232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:00.884830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.760959    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9tlh\" (UniqueName: \"kubernetes.io/projected/f0967a8b-43f1-496a-8a5c-03e435f50232-kube-api-access-p9tlh\") pod \"maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc\" (UID: \"f0967a8b-43f1-496a-8a5c-03e435f50232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:00.884830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.791108    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:00.884830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.805686    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:00.884830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.814098    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:00.884830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.845086    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:00.914313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.904918    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:00.984426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:00.984384    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:01.335026 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.334987    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"]
Nov 03 11:15:01.363503 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.363468    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"]
Nov 03 11:15:01.382758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.382656    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"]
Nov 03 11:15:01.386648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.386030    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"]
Nov 03 11:15:01.432810 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:01.432745    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda386c604_04c2_4d21_bd0d_99b4bc965c3e.slice/crio-2388f0d49c2c35df77189823e756e556f0c1aaf60a11ff6f7a0492968e8f5311 WatchSource:0}: Error finding container 2388f0d49c2c35df77189823e756e556f0c1aaf60a11ff6f7a0492968e8f5311: Status 404 returned error can't find the container with id 2388f0d49c2c35df77189823e756e556f0c1aaf60a11ff6f7a0492968e8f5311
Nov 03 11:15:01.508411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.508378    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"]
Nov 03 11:15:01.616931 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:01.616060    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da14868_435e_4ed1_96a3_b295fa1ac1a1.slice/crio-bfb604d0a4e5de4fefcf9c12967ef9ca4e075eec76189ea16fbc22d966a10cae WatchSource:0}: Error finding container bfb604d0a4e5de4fefcf9c12967ef9ca4e075eec76189ea16fbc22d966a10cae: Status 404 returned error can't find the container with id bfb604d0a4e5de4fefcf9c12967ef9ca4e075eec76189ea16fbc22d966a10cae
Nov 03 11:15:01.617903 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.617875    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"]
Nov 03 11:15:01.800047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.799985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:01.951384 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:01.951353    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbcd43e_9753_49ef_b5ad_ab1770a65cc9.slice/crio-25c43d013461bb0f59fcbe65049a05f7df3496d1a85df63e6e0fa71934d1476f WatchSource:0}: Error finding container 25c43d013461bb0f59fcbe65049a05f7df3496d1a85df63e6e0fa71934d1476f: Status 404 returned error can't find the container with id 25c43d013461bb0f59fcbe65049a05f7df3496d1a85df63e6e0fa71934d1476f
Nov 03 11:15:01.951988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.951962    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"]
Nov 03 11:15:01.974123 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:01.974091    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc01256_4c3b_40ce_9b18_7b20596c7255.slice/crio-cd5e9d3c5d1bab2635e27e500b118b008b19f89fb8ed9a88d158ab418cc0de50 WatchSource:0}: Error finding container cd5e9d3c5d1bab2635e27e500b118b008b19f89fb8ed9a88d158ab418cc0de50: Status 404 returned error can't find the container with id cd5e9d3c5d1bab2635e27e500b118b008b19f89fb8ed9a88d158ab418cc0de50
Nov 03 11:15:01.991464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:01.991269    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"]
Nov 03 11:15:02.007477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.006743    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"]
Nov 03 11:15:02.015156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.015123    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"]
Nov 03 11:15:02.060602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.060498    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" event={"ID":"63a0333b-9ed5-4e5a-95cf-ea734272085b","Type":"ContainerStarted","Data":"14cf12ee29ef4734ec75af7e0f32ac70cc08029e1137805600975a44e66821fb"}
Nov 03 11:15:02.062299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.062270    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" event={"ID":"8f21b3aa-337d-46be-ae7a-c5658f48b0af","Type":"ContainerStarted","Data":"722b30aa8c7eccb9875ac2750faaad2bf930b3f5745a2c3f5d7b0646a6355c6c"}
Nov 03 11:15:02.063851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.063826    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" event={"ID":"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9","Type":"ContainerStarted","Data":"25c43d013461bb0f59fcbe65049a05f7df3496d1a85df63e6e0fa71934d1476f"}
Nov 03 11:15:02.065325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.065297    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" event={"ID":"24da8686-666d-4c92-81fa-5e6a07e4c5bb","Type":"ContainerStarted","Data":"c71e6fc29067fa63015df4c070c2e8fe34f998677bb13d4d6c8b16028bd93b17"}
Nov 03 11:15:02.066950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.066925    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" event={"ID":"a386c604-04c2-4d21-bd0d-99b4bc965c3e","Type":"ContainerStarted","Data":"2388f0d49c2c35df77189823e756e556f0c1aaf60a11ff6f7a0492968e8f5311"}
Nov 03 11:15:02.067621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.067595    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"]
Nov 03 11:15:02.068331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.068311    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" event={"ID":"6da14868-435e-4ed1-96a3-b295fa1ac1a1","Type":"ContainerStarted","Data":"bfb604d0a4e5de4fefcf9c12967ef9ca4e075eec76189ea16fbc22d966a10cae"}
Nov 03 11:15:02.069900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.069875    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" event={"ID":"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464","Type":"ContainerStarted","Data":"7e3ac78b71d4c75152df2b1642230313638c6b507efc1409fa4bccd31245981b"}
Nov 03 11:15:02.071228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.071204    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" event={"ID":"fbc01256-4c3b-40ce-9b18-7b20596c7255","Type":"ContainerStarted","Data":"cd5e9d3c5d1bab2635e27e500b118b008b19f89fb8ed9a88d158ab418cc0de50"}
Nov 03 11:15:02.072554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.072535    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" event={"ID":"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6","Type":"ContainerStarted","Data":"2cdb00bbdf5cc91b9078fafcb7613f2f9e817fe167b73267ffe2bc5e4d854b3a"}
Nov 03 11:15:02.096786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.096761    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"]
Nov 03 11:15:02.104936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.104902    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"]
Nov 03 11:15:02.120049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.120022    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"]
Nov 03 11:15:02.151482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.151458    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"]
Nov 03 11:15:02.193831 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:02.193777    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0967a8b_43f1_496a_8a5c_03e435f50232.slice/crio-19cf371697ea24677420a7e6013f717f93abcc64fd65c7764909ffc1df672721 WatchSource:0}: Error finding container 19cf371697ea24677420a7e6013f717f93abcc64fd65c7764909ffc1df672721: Status 404 returned error can't find the container with id 19cf371697ea24677420a7e6013f717f93abcc64fd65c7764909ffc1df672721
Nov 03 11:15:02.195226 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:02.195201    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90118aeb_a009_4ce0_b6dd_81c1e08b26e9.slice/crio-ad8eebf3b0959f6fdf1ee4d78f60aa570d3ca5928d968ceb9c8aafa4fa0f79a0 WatchSource:0}: Error finding container ad8eebf3b0959f6fdf1ee4d78f60aa570d3ca5928d968ceb9c8aafa4fa0f79a0: Status 404 returned error can't find the container with id ad8eebf3b0959f6fdf1ee4d78f60aa570d3ca5928d968ceb9c8aafa4fa0f79a0
Nov 03 11:15:02.197127 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:02.197037    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a126cd_c401_485f_90f9_eaeb57262e03.slice/crio-5ff6d6025f950840b0a6d20617e35e0307cd7619da1bfb842e8e4a668e07ec29 WatchSource:0}: Error finding container 5ff6d6025f950840b0a6d20617e35e0307cd7619da1bfb842e8e4a668e07ec29: Status 404 returned error can't find the container with id 5ff6d6025f950840b0a6d20617e35e0307cd7619da1bfb842e8e4a668e07ec29
Nov 03 11:15:02.294858 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:02.294663    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38af2eca_791e_4da1_a74d_f93b4a6fa8e9.slice/crio-32f30d412a29021669c408e6b793726ef319b4cf09ddf87822481e697954a9df WatchSource:0}: Error finding container 32f30d412a29021669c408e6b793726ef319b4cf09ddf87822481e697954a9df: Status 404 returned error can't find the container with id 32f30d412a29021669c408e6b793726ef319b4cf09ddf87822481e697954a9df
Nov 03 11:15:02.483686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:02.483307    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:03.083652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.083610    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr" event={"ID":"154bc19a-4ea4-47ea-9a6a-d5c3920f902d","Type":"ContainerStarted","Data":"76f58cf0999f0d9a0f70f92b7905dfe886ab2552dee05df372f5adccaf4ac178"}
Nov 03 11:15:03.085278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.085246    5817 generic.go:356] "Generic (PLEG): container finished" podID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerID="b92257ced2aa7d9685758b33119bb8657c3c89da9ba47e15bc550e87461e34d4" exitCode=60
Nov 03 11:15:03.085412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.085330    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" event={"ID":"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6","Type":"ContainerDied","Data":"b92257ced2aa7d9685758b33119bb8657c3c89da9ba47e15bc550e87461e34d4"}
Nov 03 11:15:03.085680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.085665    5817 scope.go:122] "RemoveContainer" containerID="b92257ced2aa7d9685758b33119bb8657c3c89da9ba47e15bc550e87461e34d4"
Nov 03 11:15:03.087179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.086865    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv" event={"ID":"38af2eca-791e-4da1-a74d-f93b4a6fa8e9","Type":"ContainerStarted","Data":"32f30d412a29021669c408e6b793726ef319b4cf09ddf87822481e697954a9df"}
Nov 03 11:15:03.091731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.091701    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" event={"ID":"f0967a8b-43f1-496a-8a5c-03e435f50232","Type":"ContainerStarted","Data":"19cf371697ea24677420a7e6013f717f93abcc64fd65c7764909ffc1df672721"}
Nov 03 11:15:03.095397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.095370    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" event={"ID":"8f21b3aa-337d-46be-ae7a-c5658f48b0af","Type":"ContainerStarted","Data":"12d1480299881f6758404a7d524ef941a6528e1fc6f773162a2bd3478f38a4d3"}
Nov 03 11:15:03.118989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.118925    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" event={"ID":"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9","Type":"ContainerStarted","Data":"d844220a624f363d528ead6ae9f55d98657c7880cd9fa049c035eb4361750b73"}
Nov 03 11:15:03.128045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.127990    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" podStartSLOduration=3.127965028 podStartE2EDuration="3.127965028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:03.125196943 +0000 UTC m=+2733.330778997" watchObservedRunningTime="2026-11-03 11:15:03.127965028 +0000 UTC m=+2733.333547081"
Nov 03 11:15:03.128485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.128454    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" event={"ID":"90118aeb-a009-4ce0-b6dd-81c1e08b26e9","Type":"ContainerStarted","Data":"ad8eebf3b0959f6fdf1ee4d78f60aa570d3ca5928d968ceb9c8aafa4fa0f79a0"}
Nov 03 11:15:03.130469 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.130442    5817 generic.go:356] "Generic (PLEG): container finished" podID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerID="780dd61e0eb6ff6cf091d69a699065dd3b5e8d619e431c2915adccc51ec07cf9" exitCode=60
Nov 03 11:15:03.130592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.130487    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" event={"ID":"24da8686-666d-4c92-81fa-5e6a07e4c5bb","Type":"ContainerDied","Data":"780dd61e0eb6ff6cf091d69a699065dd3b5e8d619e431c2915adccc51ec07cf9"}
Nov 03 11:15:03.131683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.131108    5817 scope.go:122] "RemoveContainer" containerID="780dd61e0eb6ff6cf091d69a699065dd3b5e8d619e431c2915adccc51ec07cf9"
Nov 03 11:15:03.134340 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.134315    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" event={"ID":"d9a126cd-c401-485f-90f9-eaeb57262e03","Type":"ContainerStarted","Data":"5ff6d6025f950840b0a6d20617e35e0307cd7619da1bfb842e8e4a668e07ec29"}
Nov 03 11:15:03.136320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.136299    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" event={"ID":"a386c604-04c2-4d21-bd0d-99b4bc965c3e","Type":"ContainerStarted","Data":"2cc4e4976dfce4dc54acf2120b392df47120eda3dea7eea9e06919a0ef080871"}
Nov 03 11:15:03.136928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.136910    5817 scope.go:122] "RemoveContainer" containerID="2cc4e4976dfce4dc54acf2120b392df47120eda3dea7eea9e06919a0ef080871"
Nov 03 11:15:03.139001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.138969    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" event={"ID":"2d0566a2-4971-4901-a926-af80b11a0603","Type":"ContainerStarted","Data":"af4a15d17463007f4c92dd8407f3c9245117029a5254196fb8523d9bef03ad2d"}
Nov 03 11:15:03.141794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.141772    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" event={"ID":"6da14868-435e-4ed1-96a3-b295fa1ac1a1","Type":"ContainerStarted","Data":"564b309e758e49afc7afcb6a8d52ac2873ac156ef6e9d4ef907767a3a2d4d0bc"}
Nov 03 11:15:03.142133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.142115    5817 scope.go:122] "RemoveContainer" containerID="564b309e758e49afc7afcb6a8d52ac2873ac156ef6e9d4ef907767a3a2d4d0bc"
Nov 03 11:15:03.151496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.151347    5817 generic.go:356] "Generic (PLEG): container finished" podID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerID="748f2e58e8388440d6d0868b28f22e14bd63c619352426e0c5ea4241d43a4ac7" exitCode=60
Nov 03 11:15:03.151496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.151408    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" event={"ID":"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464","Type":"ContainerDied","Data":"748f2e58e8388440d6d0868b28f22e14bd63c619352426e0c5ea4241d43a4ac7"}
Nov 03 11:15:03.152473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.152072    5817 scope.go:122] "RemoveContainer" containerID="748f2e58e8388440d6d0868b28f22e14bd63c619352426e0c5ea4241d43a4ac7"
Nov 03 11:15:03.154653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.154552    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" podStartSLOduration=3.154536614 podStartE2EDuration="3.154536614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:03.147107474 +0000 UTC m=+2733.352689526" watchObservedRunningTime="2026-11-03 11:15:03.154536614 +0000 UTC m=+2733.360118669"
Nov 03 11:15:03.323656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:03.320848    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:04.196041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.195877    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:04.251490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.251319    5817 generic.go:356] "Generic (PLEG): container finished" podID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerID="d844220a624f363d528ead6ae9f55d98657c7880cd9fa049c035eb4361750b73" exitCode=60
Nov 03 11:15:04.251490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.251368    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" event={"ID":"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9","Type":"ContainerDied","Data":"d844220a624f363d528ead6ae9f55d98657c7880cd9fa049c035eb4361750b73"}
Nov 03 11:15:04.251899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.251876    5817 scope.go:122] "RemoveContainer" containerID="d844220a624f363d528ead6ae9f55d98657c7880cd9fa049c035eb4361750b73"
Nov 03 11:15:04.255840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.255810    5817 generic.go:356] "Generic (PLEG): container finished" podID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerID="616233aed5fa3b3fa5ed5127a4a3d88464286728d6acb53552bea7d61072151a" exitCode=60
Nov 03 11:15:04.256025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.255899    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" event={"ID":"90118aeb-a009-4ce0-b6dd-81c1e08b26e9","Type":"ContainerDied","Data":"616233aed5fa3b3fa5ed5127a4a3d88464286728d6acb53552bea7d61072151a"}
Nov 03 11:15:04.256340 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.256317    5817 scope.go:122] "RemoveContainer" containerID="616233aed5fa3b3fa5ed5127a4a3d88464286728d6acb53552bea7d61072151a"
Nov 03 11:15:04.263685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.259506    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" event={"ID":"24da8686-666d-4c92-81fa-5e6a07e4c5bb","Type":"ContainerStarted","Data":"34d5bb78a3eb4132130bcf307508e1eb3152b8de9238d9740d50b76ec58d17d0"}
Nov 03 11:15:04.264313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.264288    5817 generic.go:356] "Generic (PLEG): container finished" podID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerID="54ae9bdf7b6a4de0ddc62195d2b989e436e851ec7a571daa3631711790111317" exitCode=60
Nov 03 11:15:04.264456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.264364    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" event={"ID":"d9a126cd-c401-485f-90f9-eaeb57262e03","Type":"ContainerDied","Data":"54ae9bdf7b6a4de0ddc62195d2b989e436e851ec7a571daa3631711790111317"}
Nov 03 11:15:04.264650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.264634    5817 scope.go:122] "RemoveContainer" containerID="54ae9bdf7b6a4de0ddc62195d2b989e436e851ec7a571daa3631711790111317"
Nov 03 11:15:04.269273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.269248    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" event={"ID":"a386c604-04c2-4d21-bd0d-99b4bc965c3e","Type":"ContainerDied","Data":"2cc4e4976dfce4dc54acf2120b392df47120eda3dea7eea9e06919a0ef080871"}
Nov 03 11:15:04.271632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.269855    5817 generic.go:356] "Generic (PLEG): container finished" podID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerID="2cc4e4976dfce4dc54acf2120b392df47120eda3dea7eea9e06919a0ef080871" exitCode=60
Nov 03 11:15:04.271632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.269961    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" event={"ID":"a386c604-04c2-4d21-bd0d-99b4bc965c3e","Type":"ContainerStarted","Data":"b68e28287ad3c5c849428532de64b74810b1de822a817b65664486db8423be34"}
Nov 03 11:15:04.273461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.273315    5817 generic.go:356] "Generic (PLEG): container finished" podID="2d0566a2-4971-4901-a926-af80b11a0603" containerID="8a55c6a70d1baf619b95a45a356522944250786ac0e79cc75a42655aa94c43b8" exitCode=60
Nov 03 11:15:04.273461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.273420    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" event={"ID":"2d0566a2-4971-4901-a926-af80b11a0603","Type":"ContainerDied","Data":"8a55c6a70d1baf619b95a45a356522944250786ac0e79cc75a42655aa94c43b8"}
Nov 03 11:15:04.273691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.273618    5817 scope.go:122] "RemoveContainer" containerID="8a55c6a70d1baf619b95a45a356522944250786ac0e79cc75a42655aa94c43b8"
Nov 03 11:15:04.276710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.276687    5817 generic.go:356] "Generic (PLEG): container finished" podID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerID="564b309e758e49afc7afcb6a8d52ac2873ac156ef6e9d4ef907767a3a2d4d0bc" exitCode=60
Nov 03 11:15:04.276821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.276766    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" event={"ID":"6da14868-435e-4ed1-96a3-b295fa1ac1a1","Type":"ContainerDied","Data":"564b309e758e49afc7afcb6a8d52ac2873ac156ef6e9d4ef907767a3a2d4d0bc"}
Nov 03 11:15:04.276821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.276804    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" event={"ID":"6da14868-435e-4ed1-96a3-b295fa1ac1a1","Type":"ContainerStarted","Data":"ce66cd30365b159f84c15eb53a330d4bd5fea6f803ac22066d942ba7d15c52fa"}
Nov 03 11:15:04.279479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.279455    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" event={"ID":"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464","Type":"ContainerStarted","Data":"162dbd6044746dc8911734263add382fc3ef556dce4d381b946e25d4c2d57bea"}
Nov 03 11:15:04.281114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.281092    5817 generic.go:356] "Generic (PLEG): container finished" podID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerID="b982b55c0243b7d1dc38606b0fb50e74d32ead91720f38af44c991c50775ae03" exitCode=60
Nov 03 11:15:04.281230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.281173    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" event={"ID":"fbc01256-4c3b-40ce-9b18-7b20596c7255","Type":"ContainerDied","Data":"b982b55c0243b7d1dc38606b0fb50e74d32ead91720f38af44c991c50775ae03"}
Nov 03 11:15:04.281363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.281345    5817 scope.go:122] "RemoveContainer" containerID="b982b55c0243b7d1dc38606b0fb50e74d32ead91720f38af44c991c50775ae03"
Nov 03 11:15:04.285919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.285895    5817 generic.go:356] "Generic (PLEG): container finished" podID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerID="a020faefe35581f3d54ea59a38c763547896517740152e1d4337be3bf5ecbc24" exitCode=60
Nov 03 11:15:04.286032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.285998    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr" event={"ID":"154bc19a-4ea4-47ea-9a6a-d5c3920f902d","Type":"ContainerDied","Data":"a020faefe35581f3d54ea59a38c763547896517740152e1d4337be3bf5ecbc24"}
Nov 03 11:15:04.286487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.286183    5817 scope.go:122] "RemoveContainer" containerID="a020faefe35581f3d54ea59a38c763547896517740152e1d4337be3bf5ecbc24"
Nov 03 11:15:04.290700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.289513    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" event={"ID":"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6","Type":"ContainerStarted","Data":"1f064800da004087287bef7c07bf782647701b3fdd1f5dd056de4f801c7297cd"}
Nov 03 11:15:04.296799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.296763    5817 generic.go:356] "Generic (PLEG): container finished" podID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerID="aebee4cae524953d7bd72747310cc3a20f9bef867800bcd9b62c04f8c95cb76c" exitCode=60
Nov 03 11:15:04.296911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.296872    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv" event={"ID":"38af2eca-791e-4da1-a74d-f93b4a6fa8e9","Type":"ContainerDied","Data":"aebee4cae524953d7bd72747310cc3a20f9bef867800bcd9b62c04f8c95cb76c"}
Nov 03 11:15:04.297894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.297259    5817 scope.go:122] "RemoveContainer" containerID="aebee4cae524953d7bd72747310cc3a20f9bef867800bcd9b62c04f8c95cb76c"
Nov 03 11:15:04.319438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.318608    5817 generic.go:356] "Generic (PLEG): container finished" podID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerID="c7c1ee645c9507c832d0351287da54ad1dacc68d532314b8f13defadaa4f4f91" exitCode=60
Nov 03 11:15:04.319438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.318711    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" event={"ID":"f0967a8b-43f1-496a-8a5c-03e435f50232","Type":"ContainerDied","Data":"c7c1ee645c9507c832d0351287da54ad1dacc68d532314b8f13defadaa4f4f91"}
Nov 03 11:15:04.319438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.319050    5817 scope.go:122] "RemoveContainer" containerID="c7c1ee645c9507c832d0351287da54ad1dacc68d532314b8f13defadaa4f4f91"
Nov 03 11:15:04.322690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.322658    5817 generic.go:356] "Generic (PLEG): container finished" podID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerID="336f8a94ee3ef5a9244a3c995311c95cc9867811b43a2cb1037fc4cd65ea35d6" exitCode=60
Nov 03 11:15:04.322808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.322760    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" event={"ID":"63a0333b-9ed5-4e5a-95cf-ea734272085b","Type":"ContainerDied","Data":"336f8a94ee3ef5a9244a3c995311c95cc9867811b43a2cb1037fc4cd65ea35d6"}
Nov 03 11:15:04.323326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.323305    5817 scope.go:122] "RemoveContainer" containerID="336f8a94ee3ef5a9244a3c995311c95cc9867811b43a2cb1037fc4cd65ea35d6"
Nov 03 11:15:04.323394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.323352    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" podStartSLOduration=4.323338666 podStartE2EDuration="4.323338666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:04.317806087 +0000 UTC m=+2734.523388139" watchObservedRunningTime="2026-11-03 11:15:04.323338666 +0000 UTC m=+2734.528920717"
Nov 03 11:15:04.339908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.339878    5817 generic.go:356] "Generic (PLEG): container finished" podID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerID="12d1480299881f6758404a7d524ef941a6528e1fc6f773162a2bd3478f38a4d3" exitCode=60
Nov 03 11:15:04.340048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.339933    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" event={"ID":"8f21b3aa-337d-46be-ae7a-c5658f48b0af","Type":"ContainerDied","Data":"12d1480299881f6758404a7d524ef941a6528e1fc6f773162a2bd3478f38a4d3"}
Nov 03 11:15:04.340974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.340590    5817 scope.go:122] "RemoveContainer" containerID="12d1480299881f6758404a7d524ef941a6528e1fc6f773162a2bd3478f38a4d3"
Nov 03 11:15:04.379602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.373165    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" podStartSLOduration=4.37314557 podStartE2EDuration="4.37314557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:04.337785681 +0000 UTC m=+2734.543367733" watchObservedRunningTime="2026-11-03 11:15:04.37314557 +0000 UTC m=+2734.578727621"
Nov 03 11:15:04.423033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.422700    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" podStartSLOduration=4.422680724 podStartE2EDuration="4.422680724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:04.414828164 +0000 UTC m=+2734.620410214" watchObservedRunningTime="2026-11-03 11:15:04.422680724 +0000 UTC m=+2734.628262775"
Nov 03 11:15:04.447197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.447136    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" podStartSLOduration=4.447117678 podStartE2EDuration="4.447117678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:04.445541593 +0000 UTC m=+2734.651123643" watchObservedRunningTime="2026-11-03 11:15:04.447117678 +0000 UTC m=+2734.652699730"
Nov 03 11:15:04.464215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:04.464163    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" podStartSLOduration=4.464143087 podStartE2EDuration="4.464143087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:04.460677902 +0000 UTC m=+2734.666259962" watchObservedRunningTime="2026-11-03 11:15:04.464143087 +0000 UTC m=+2734.669725177"
Nov 03 11:15:05.079612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.079502    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:05.356360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.356262    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" event={"ID":"90118aeb-a009-4ce0-b6dd-81c1e08b26e9","Type":"ContainerStarted","Data":"b8a932f31e3d50ff3723bfad47c16ce0f9da6ad746aa4a26961dbd4181d749e8"}
Nov 03 11:15:05.358895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.358866    5817 generic.go:356] "Generic (PLEG): container finished" podID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerID="34d5bb78a3eb4132130bcf307508e1eb3152b8de9238d9740d50b76ec58d17d0" exitCode=60
Nov 03 11:15:05.359020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.358942    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" event={"ID":"24da8686-666d-4c92-81fa-5e6a07e4c5bb","Type":"ContainerDied","Data":"34d5bb78a3eb4132130bcf307508e1eb3152b8de9238d9740d50b76ec58d17d0"}
Nov 03 11:15:05.359020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.359005    5817 scope.go:122] "RemoveContainer" containerID="780dd61e0eb6ff6cf091d69a699065dd3b5e8d619e431c2915adccc51ec07cf9"
Nov 03 11:15:05.359305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.359288    5817 scope.go:122] "RemoveContainer" containerID="34d5bb78a3eb4132130bcf307508e1eb3152b8de9238d9740d50b76ec58d17d0"
Nov 03 11:15:05.359838 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:05.359811    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss_odh-ai-gateway-infra(24da8686-666d-4c92-81fa-5e6a07e4c5bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb"
Nov 03 11:15:05.361162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.361139    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" event={"ID":"d9a126cd-c401-485f-90f9-eaeb57262e03","Type":"ContainerStarted","Data":"d68a3b1f7553b4bb15742f6d21b77564f9223e67f96760d1ffd37c748d0498d9"}
Nov 03 11:15:05.363473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.363449    5817 generic.go:356] "Generic (PLEG): container finished" podID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerID="b68e28287ad3c5c849428532de64b74810b1de822a817b65664486db8423be34" exitCode=60
Nov 03 11:15:05.363606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.363528    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" event={"ID":"a386c604-04c2-4d21-bd0d-99b4bc965c3e","Type":"ContainerDied","Data":"b68e28287ad3c5c849428532de64b74810b1de822a817b65664486db8423be34"}
Nov 03 11:15:05.363846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.363828    5817 scope.go:122] "RemoveContainer" containerID="b68e28287ad3c5c849428532de64b74810b1de822a817b65664486db8423be34"
Nov 03 11:15:05.364181 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:05.364161    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs_odh-ai-gateway-infra(a386c604-04c2-4d21-bd0d-99b4bc965c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e"
Nov 03 11:15:05.365871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.365849    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" event={"ID":"2d0566a2-4971-4901-a926-af80b11a0603","Type":"ContainerStarted","Data":"40acc8fa1797b6ad6bdad7dba266d1fcde7dcd649a073330cc7308dfce85e1b2"}
Nov 03 11:15:05.368189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.368170    5817 generic.go:356] "Generic (PLEG): container finished" podID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerID="ce66cd30365b159f84c15eb53a330d4bd5fea6f803ac22066d942ba7d15c52fa" exitCode=60
Nov 03 11:15:05.368304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.368235    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" event={"ID":"6da14868-435e-4ed1-96a3-b295fa1ac1a1","Type":"ContainerDied","Data":"ce66cd30365b159f84c15eb53a330d4bd5fea6f803ac22066d942ba7d15c52fa"}
Nov 03 11:15:05.368558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.368542    5817 scope.go:122] "RemoveContainer" containerID="ce66cd30365b159f84c15eb53a330d4bd5fea6f803ac22066d942ba7d15c52fa"
Nov 03 11:15:05.368866 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:05.368845    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk_odh-ai-gateway-infra(6da14868-435e-4ed1-96a3-b295fa1ac1a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1"
Nov 03 11:15:05.370128 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.370108    5817 generic.go:356] "Generic (PLEG): container finished" podID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerID="162dbd6044746dc8911734263add382fc3ef556dce4d381b946e25d4c2d57bea" exitCode=60
Nov 03 11:15:05.370213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.370191    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" event={"ID":"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464","Type":"ContainerDied","Data":"162dbd6044746dc8911734263add382fc3ef556dce4d381b946e25d4c2d57bea"}
Nov 03 11:15:05.370445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.370419    5817 scope.go:122] "RemoveContainer" containerID="162dbd6044746dc8911734263add382fc3ef556dce4d381b946e25d4c2d57bea"
Nov 03 11:15:05.370743 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:05.370719    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr_odh-ai-gateway-infra(72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464"
Nov 03 11:15:05.373076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.373054    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" event={"ID":"fbc01256-4c3b-40ce-9b18-7b20596c7255","Type":"ContainerStarted","Data":"d55f8648d464735f54e7c369a9548f10a9ab94222c43d988fe31c7c238f0b769"}
Nov 03 11:15:05.376046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.376024    5817 generic.go:356] "Generic (PLEG): container finished" podID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerID="1f064800da004087287bef7c07bf782647701b3fdd1f5dd056de4f801c7297cd" exitCode=60
Nov 03 11:15:05.376158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.376109    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" event={"ID":"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6","Type":"ContainerDied","Data":"1f064800da004087287bef7c07bf782647701b3fdd1f5dd056de4f801c7297cd"}
Nov 03 11:15:05.376321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.376302    5817 scope.go:122] "RemoveContainer" containerID="1f064800da004087287bef7c07bf782647701b3fdd1f5dd056de4f801c7297cd"
Nov 03 11:15:05.376629 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:05.376603    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5_odh-ai-gateway-infra(55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6"
Nov 03 11:15:05.377124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.377084    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" podStartSLOduration=5.377069722 podStartE2EDuration="5.377069722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:05.372493825 +0000 UTC m=+2735.578075899" watchObservedRunningTime="2026-11-03 11:15:05.377069722 +0000 UTC m=+2735.582651774"
Nov 03 11:15:05.378473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.378440    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" event={"ID":"f0967a8b-43f1-496a-8a5c-03e435f50232","Type":"ContainerStarted","Data":"9ebc224dd1e01dd02211bdfd6968714e3cf139ecd36e14d743da6085fcc8717b"}
Nov 03 11:15:05.380638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.380600    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" event={"ID":"63a0333b-9ed5-4e5a-95cf-ea734272085b","Type":"ContainerStarted","Data":"537e0537e0b4f0245c0d8baaacde49027b04d85db0b93642674960b94cce35e4"}
Nov 03 11:15:05.383073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.383050    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" event={"ID":"8f21b3aa-337d-46be-ae7a-c5658f48b0af","Type":"ContainerStarted","Data":"091f04ecca72591f4a35adf003b4ac9197205e2f50d14e241d291fe1af15201b"}
Nov 03 11:15:05.386162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.386136    5817 scope.go:122] "RemoveContainer" containerID="2cc4e4976dfce4dc54acf2120b392df47120eda3dea7eea9e06919a0ef080871"
Nov 03 11:15:05.399068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.398623    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" podStartSLOduration=5.398607511 podStartE2EDuration="5.398607511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:05.390472403 +0000 UTC m=+2735.596054456" watchObservedRunningTime="2026-11-03 11:15:05.398607511 +0000 UTC m=+2735.604189562"
Nov 03 11:15:05.411276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.411258    5817 scope.go:122] "RemoveContainer" containerID="564b309e758e49afc7afcb6a8d52ac2873ac156ef6e9d4ef907767a3a2d4d0bc"
Nov 03 11:15:05.426662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.426608    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" podStartSLOduration=5.426590315 podStartE2EDuration="5.426590315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:05.425119365 +0000 UTC m=+2735.630701429" watchObservedRunningTime="2026-11-03 11:15:05.426590315 +0000 UTC m=+2735.632172392"
Nov 03 11:15:05.429236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.429181    5817 scope.go:122] "RemoveContainer" containerID="748f2e58e8388440d6d0868b28f22e14bd63c619352426e0c5ea4241d43a4ac7"
Nov 03 11:15:05.448910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.448883    5817 scope.go:122] "RemoveContainer" containerID="b92257ced2aa7d9685758b33119bb8657c3c89da9ba47e15bc550e87461e34d4"
Nov 03 11:15:05.493839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.492801    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" podStartSLOduration=5.492783651 podStartE2EDuration="5.492783651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:05.492440835 +0000 UTC m=+2735.698022887" watchObservedRunningTime="2026-11-03 11:15:05.492783651 +0000 UTC m=+2735.698365702"
Nov 03 11:15:05.512859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.512801    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" podStartSLOduration=5.512787868 podStartE2EDuration="5.512787868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:05.506823341 +0000 UTC m=+2735.712405394" watchObservedRunningTime="2026-11-03 11:15:05.512787868 +0000 UTC m=+2735.718369938"
Nov 03 11:15:05.527184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.527133    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" podStartSLOduration=5.527119905 podStartE2EDuration="5.527119905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:05.521803006 +0000 UTC m=+2735.727385060" watchObservedRunningTime="2026-11-03 11:15:05.527119905 +0000 UTC m=+2735.732701956"
Nov 03 11:15:05.950262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:05.950186    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:06.398014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.397979    5817 generic.go:356] "Generic (PLEG): container finished" podID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerID="9ebc224dd1e01dd02211bdfd6968714e3cf139ecd36e14d743da6085fcc8717b" exitCode=60
Nov 03 11:15:06.398447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.398030    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" event={"ID":"f0967a8b-43f1-496a-8a5c-03e435f50232","Type":"ContainerDied","Data":"9ebc224dd1e01dd02211bdfd6968714e3cf139ecd36e14d743da6085fcc8717b"}
Nov 03 11:15:06.398447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.398136    5817 scope.go:122] "RemoveContainer" containerID="c7c1ee645c9507c832d0351287da54ad1dacc68d532314b8f13defadaa4f4f91"
Nov 03 11:15:06.398748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.398733    5817 scope.go:122] "RemoveContainer" containerID="9ebc224dd1e01dd02211bdfd6968714e3cf139ecd36e14d743da6085fcc8717b"
Nov 03 11:15:06.399450 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.399164    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc_odh-ai-gateway-infra(f0967a8b-43f1-496a-8a5c-03e435f50232)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232"
Nov 03 11:15:06.400372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.400352    5817 generic.go:356] "Generic (PLEG): container finished" podID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerID="537e0537e0b4f0245c0d8baaacde49027b04d85db0b93642674960b94cce35e4" exitCode=60
Nov 03 11:15:06.400459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.400400    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" event={"ID":"63a0333b-9ed5-4e5a-95cf-ea734272085b","Type":"ContainerDied","Data":"537e0537e0b4f0245c0d8baaacde49027b04d85db0b93642674960b94cce35e4"}
Nov 03 11:15:06.400811 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.400789    5817 scope.go:122] "RemoveContainer" containerID="537e0537e0b4f0245c0d8baaacde49027b04d85db0b93642674960b94cce35e4"
Nov 03 11:15:06.401262 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.401240    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c_odh-ai-gateway-infra(63a0333b-9ed5-4e5a-95cf-ea734272085b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b"
Nov 03 11:15:06.402607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.402585    5817 generic.go:356] "Generic (PLEG): container finished" podID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerID="091f04ecca72591f4a35adf003b4ac9197205e2f50d14e241d291fe1af15201b" exitCode=60
Nov 03 11:15:06.402705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.402665    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" event={"ID":"8f21b3aa-337d-46be-ae7a-c5658f48b0af","Type":"ContainerDied","Data":"091f04ecca72591f4a35adf003b4ac9197205e2f50d14e241d291fe1af15201b"}
Nov 03 11:15:06.402988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.402965    5817 scope.go:122] "RemoveContainer" containerID="091f04ecca72591f4a35adf003b4ac9197205e2f50d14e241d291fe1af15201b"
Nov 03 11:15:06.403326 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.403307    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc_odh-ai-gateway-infra(8f21b3aa-337d-46be-ae7a-c5658f48b0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af"
Nov 03 11:15:06.405057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.405031    5817 generic.go:356] "Generic (PLEG): container finished" podID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerID="0e5e1dc6aece02b659dc54fcb403ff720e01d6bcc5d5736f53bf7cdb3f7d9279" exitCode=60
Nov 03 11:15:06.405153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.405108    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" event={"ID":"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9","Type":"ContainerDied","Data":"0e5e1dc6aece02b659dc54fcb403ff720e01d6bcc5d5736f53bf7cdb3f7d9279"}
Nov 03 11:15:06.405469 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.405449    5817 scope.go:122] "RemoveContainer" containerID="0e5e1dc6aece02b659dc54fcb403ff720e01d6bcc5d5736f53bf7cdb3f7d9279"
Nov 03 11:15:06.405819 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.405796    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895075-tvldd_odh-ai-gateway-infra(bcbcd43e-9753-49ef-b5ad-ab1770a65cc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9"
Nov 03 11:15:06.407192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.407174    5817 generic.go:356] "Generic (PLEG): container finished" podID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerID="b8a932f31e3d50ff3723bfad47c16ce0f9da6ad746aa4a26961dbd4181d749e8" exitCode=60
Nov 03 11:15:06.407298 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.407241    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" event={"ID":"90118aeb-a009-4ce0-b6dd-81c1e08b26e9","Type":"ContainerDied","Data":"b8a932f31e3d50ff3723bfad47c16ce0f9da6ad746aa4a26961dbd4181d749e8"}
Nov 03 11:15:06.407534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.407519    5817 scope.go:122] "RemoveContainer" containerID="b8a932f31e3d50ff3723bfad47c16ce0f9da6ad746aa4a26961dbd4181d749e8"
Nov 03 11:15:06.407934 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.407895    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp_odh-ai-gateway-infra(90118aeb-a009-4ce0-b6dd-81c1e08b26e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9"
Nov 03 11:15:06.409354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.409336    5817 scope.go:122] "RemoveContainer" containerID="34d5bb78a3eb4132130bcf307508e1eb3152b8de9238d9740d50b76ec58d17d0"
Nov 03 11:15:06.409661 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.409635    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss_odh-ai-gateway-infra(24da8686-666d-4c92-81fa-5e6a07e4c5bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb"
Nov 03 11:15:06.411096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.411076    5817 generic.go:356] "Generic (PLEG): container finished" podID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerID="d68a3b1f7553b4bb15742f6d21b77564f9223e67f96760d1ffd37c748d0498d9" exitCode=60
Nov 03 11:15:06.411178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.411142    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" event={"ID":"d9a126cd-c401-485f-90f9-eaeb57262e03","Type":"ContainerDied","Data":"d68a3b1f7553b4bb15742f6d21b77564f9223e67f96760d1ffd37c748d0498d9"}
Nov 03 11:15:06.411366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.411351    5817 scope.go:122] "RemoveContainer" containerID="d68a3b1f7553b4bb15742f6d21b77564f9223e67f96760d1ffd37c748d0498d9"
Nov 03 11:15:06.412412 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.412388    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl_odh-ai-gateway-infra(d9a126cd-c401-485f-90f9-eaeb57262e03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03"
Nov 03 11:15:06.413384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.413358    5817 scope.go:122] "RemoveContainer" containerID="b68e28287ad3c5c849428532de64b74810b1de822a817b65664486db8423be34"
Nov 03 11:15:06.413748 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.413677    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs_odh-ai-gateway-infra(a386c604-04c2-4d21-bd0d-99b4bc965c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e"
Nov 03 11:15:06.415740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.415717    5817 generic.go:356] "Generic (PLEG): container finished" podID="2d0566a2-4971-4901-a926-af80b11a0603" containerID="40acc8fa1797b6ad6bdad7dba266d1fcde7dcd649a073330cc7308dfce85e1b2" exitCode=60
Nov 03 11:15:06.415830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.415791    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" event={"ID":"2d0566a2-4971-4901-a926-af80b11a0603","Type":"ContainerDied","Data":"40acc8fa1797b6ad6bdad7dba266d1fcde7dcd649a073330cc7308dfce85e1b2"}
Nov 03 11:15:06.416024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.416008    5817 scope.go:122] "RemoveContainer" containerID="40acc8fa1797b6ad6bdad7dba266d1fcde7dcd649a073330cc7308dfce85e1b2"
Nov 03 11:15:06.416288 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.416267    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd_odh-ai-gateway-infra(2d0566a2-4971-4901-a926-af80b11a0603)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" podUID="2d0566a2-4971-4901-a926-af80b11a0603"
Nov 03 11:15:06.417959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.417940    5817 scope.go:122] "RemoveContainer" containerID="ce66cd30365b159f84c15eb53a330d4bd5fea6f803ac22066d942ba7d15c52fa"
Nov 03 11:15:06.418289 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.418213    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk_odh-ai-gateway-infra(6da14868-435e-4ed1-96a3-b295fa1ac1a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1"
Nov 03 11:15:06.419832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.419815    5817 scope.go:122] "RemoveContainer" containerID="162dbd6044746dc8911734263add382fc3ef556dce4d381b946e25d4c2d57bea"
Nov 03 11:15:06.420079 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.420057    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr_odh-ai-gateway-infra(72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464"
Nov 03 11:15:06.421903 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.421885    5817 generic.go:356] "Generic (PLEG): container finished" podID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerID="d55f8648d464735f54e7c369a9548f10a9ab94222c43d988fe31c7c238f0b769" exitCode=60
Nov 03 11:15:06.421975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.421915    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" event={"ID":"fbc01256-4c3b-40ce-9b18-7b20596c7255","Type":"ContainerDied","Data":"d55f8648d464735f54e7c369a9548f10a9ab94222c43d988fe31c7c238f0b769"}
Nov 03 11:15:06.422228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.422211    5817 scope.go:122] "RemoveContainer" containerID="d55f8648d464735f54e7c369a9548f10a9ab94222c43d988fe31c7c238f0b769"
Nov 03 11:15:06.422530 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.422507    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t_odh-ai-gateway-infra(fbc01256-4c3b-40ce-9b18-7b20596c7255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255"
Nov 03 11:15:06.424861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.424651    5817 generic.go:356] "Generic (PLEG): container finished" podID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerID="c669e591d6f743cfd32489da19c65980f911c5cf5f2c29c23bc3853e1522d3ce" exitCode=60
Nov 03 11:15:06.424861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.424751    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr" event={"ID":"154bc19a-4ea4-47ea-9a6a-d5c3920f902d","Type":"ContainerDied","Data":"c669e591d6f743cfd32489da19c65980f911c5cf5f2c29c23bc3853e1522d3ce"}
Nov 03 11:15:06.425049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.424978    5817 scope.go:122] "RemoveContainer" containerID="c669e591d6f743cfd32489da19c65980f911c5cf5f2c29c23bc3853e1522d3ce"
Nov 03 11:15:06.425245 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.425223    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr_odh-ai-gateway-infra(154bc19a-4ea4-47ea-9a6a-d5c3920f902d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d"
Nov 03 11:15:06.426841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.426825    5817 scope.go:122] "RemoveContainer" containerID="1f064800da004087287bef7c07bf782647701b3fdd1f5dd056de4f801c7297cd"
Nov 03 11:15:06.427051 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.427030    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5_odh-ai-gateway-infra(55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6"
Nov 03 11:15:06.429829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.429806    5817 generic.go:356] "Generic (PLEG): container finished" podID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerID="1e49ca6e8f3b20855382a02523bef8bf86a27d026d95e4d1bd033461aee2e22b" exitCode=60
Nov 03 11:15:06.429915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.429851    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv" event={"ID":"38af2eca-791e-4da1-a74d-f93b4a6fa8e9","Type":"ContainerDied","Data":"1e49ca6e8f3b20855382a02523bef8bf86a27d026d95e4d1bd033461aee2e22b"}
Nov 03 11:15:06.430191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.430175    5817 scope.go:122] "RemoveContainer" containerID="1e49ca6e8f3b20855382a02523bef8bf86a27d026d95e4d1bd033461aee2e22b"
Nov 03 11:15:06.430468 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:06.430446    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv_odh-ai-gateway-infra(38af2eca-791e-4da1-a74d-f93b4a6fa8e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9"
Nov 03 11:15:06.459231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.459206    5817 scope.go:122] "RemoveContainer" containerID="336f8a94ee3ef5a9244a3c995311c95cc9867811b43a2cb1037fc4cd65ea35d6"
Nov 03 11:15:06.475363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.475343    5817 scope.go:122] "RemoveContainer" containerID="12d1480299881f6758404a7d524ef941a6528e1fc6f773162a2bd3478f38a4d3"
Nov 03 11:15:06.491723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.491700    5817 scope.go:122] "RemoveContainer" containerID="d844220a624f363d528ead6ae9f55d98657c7880cd9fa049c035eb4361750b73"
Nov 03 11:15:06.509734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.509708    5817 scope.go:122] "RemoveContainer" containerID="616233aed5fa3b3fa5ed5127a4a3d88464286728d6acb53552bea7d61072151a"
Nov 03 11:15:06.530185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.530160    5817 scope.go:122] "RemoveContainer" containerID="54ae9bdf7b6a4de0ddc62195d2b989e436e851ec7a571daa3631711790111317"
Nov 03 11:15:06.548844 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.548822    5817 scope.go:122] "RemoveContainer" containerID="8a55c6a70d1baf619b95a45a356522944250786ac0e79cc75a42655aa94c43b8"
Nov 03 11:15:06.607608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.604116    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:06.656789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.656631    5817 scope.go:122] "RemoveContainer" containerID="b982b55c0243b7d1dc38606b0fb50e74d32ead91720f38af44c991c50775ae03"
Nov 03 11:15:06.679772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.679744    5817 scope.go:122] "RemoveContainer" containerID="a020faefe35581f3d54ea59a38c763547896517740152e1d4337be3bf5ecbc24"
Nov 03 11:15:06.699268 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.699245    5817 scope.go:122] "RemoveContainer" containerID="aebee4cae524953d7bd72747310cc3a20f9bef867800bcd9b62c04f8c95cb76c"
Nov 03 11:15:06.829510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:06.829339    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:15:07.449260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.449236    5817 scope.go:122] "RemoveContainer" containerID="1e49ca6e8f3b20855382a02523bef8bf86a27d026d95e4d1bd033461aee2e22b"
Nov 03 11:15:07.450245 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.450201    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv_odh-ai-gateway-infra(38af2eca-791e-4da1-a74d-f93b4a6fa8e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9"
Nov 03 11:15:07.453403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.453304    5817 scope.go:122] "RemoveContainer" containerID="9ebc224dd1e01dd02211bdfd6968714e3cf139ecd36e14d743da6085fcc8717b"
Nov 03 11:15:07.454106 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.454076    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc_odh-ai-gateway-infra(f0967a8b-43f1-496a-8a5c-03e435f50232)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232"
Nov 03 11:15:07.460928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.460906    5817 scope.go:122] "RemoveContainer" containerID="537e0537e0b4f0245c0d8baaacde49027b04d85db0b93642674960b94cce35e4"
Nov 03 11:15:07.461863 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.461549    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c_odh-ai-gateway-infra(63a0333b-9ed5-4e5a-95cf-ea734272085b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b"
Nov 03 11:15:07.465335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.465318    5817 scope.go:122] "RemoveContainer" containerID="091f04ecca72591f4a35adf003b4ac9197205e2f50d14e241d291fe1af15201b"
Nov 03 11:15:07.465990 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.465960    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc_odh-ai-gateway-infra(8f21b3aa-337d-46be-ae7a-c5658f48b0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af"
Nov 03 11:15:07.475559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.475535    5817 scope.go:122] "RemoveContainer" containerID="0e5e1dc6aece02b659dc54fcb403ff720e01d6bcc5d5736f53bf7cdb3f7d9279"
Nov 03 11:15:07.476186 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.476156    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895075-tvldd_odh-ai-gateway-infra(bcbcd43e-9753-49ef-b5ad-ab1770a65cc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9"
Nov 03 11:15:07.480746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.480726    5817 scope.go:122] "RemoveContainer" containerID="b8a932f31e3d50ff3723bfad47c16ce0f9da6ad746aa4a26961dbd4181d749e8"
Nov 03 11:15:07.482263 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.481556    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp_odh-ai-gateway-infra(90118aeb-a009-4ce0-b6dd-81c1e08b26e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9"
Nov 03 11:15:07.485594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.485239    5817 scope.go:122] "RemoveContainer" containerID="d68a3b1f7553b4bb15742f6d21b77564f9223e67f96760d1ffd37c748d0498d9"
Nov 03 11:15:07.487242 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.485693    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl_odh-ai-gateway-infra(d9a126cd-c401-485f-90f9-eaeb57262e03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03"
Nov 03 11:15:07.492470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.491596    5817 scope.go:122] "RemoveContainer" containerID="40acc8fa1797b6ad6bdad7dba266d1fcde7dcd649a073330cc7308dfce85e1b2"
Nov 03 11:15:07.492470 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.491847    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd_odh-ai-gateway-infra(2d0566a2-4971-4901-a926-af80b11a0603)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" podUID="2d0566a2-4971-4901-a926-af80b11a0603"
Nov 03 11:15:07.497065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.497049    5817 scope.go:122] "RemoveContainer" containerID="d55f8648d464735f54e7c369a9548f10a9ab94222c43d988fe31c7c238f0b769"
Nov 03 11:15:07.497453 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.497432    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t_odh-ai-gateway-infra(fbc01256-4c3b-40ce-9b18-7b20596c7255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255"
Nov 03 11:15:07.502748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:07.502728    5817 scope.go:122] "RemoveContainer" containerID="c669e591d6f743cfd32489da19c65980f911c5cf5f2c29c23bc3853e1522d3ce"
Nov 03 11:15:07.503103 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:07.503080    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr_odh-ai-gateway-infra(154bc19a-4ea4-47ea-9a6a-d5c3920f902d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d"
Nov 03 11:15:08.549470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:08.549431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:08.683731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:08.683695    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:09.270594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:09.270542    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:09.980643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:09.980609    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:10.809614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:10.805271    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:11.259602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:11.237035    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:11.540664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:11.537203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:11.708372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:11.708340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:12.148403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.148369    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"]
Nov 03 11:15:12.154950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.154925    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.169565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.166983    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3eb43f-openshift-default-dockercfg-vc686\""
Nov 03 11:15:12.169565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.169391    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"]
Nov 03 11:15:12.237562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.237417    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:12.312221 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.312192    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea678e2-cfc3-4ff5-8889-601be2dd2909-istiod-ca-cert\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.312677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.312235    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zjm\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-kube-api-access-25zjm\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.312677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.312405    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-data\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.312677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.312456    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-socket\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.312677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.312552    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-certs\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.312677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.312660    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-envoy\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.312945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.312709    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-podinfo\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.312945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.312901    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-credential-socket\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.313055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.313035    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-token\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.408657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.402519    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:12.423602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.423105    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-certs\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.423602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.422554    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-certs\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.423602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.423367    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-envoy\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.423925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.423895    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-podinfo\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.431596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.428818    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-credential-socket\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.431596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.428941    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-token\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.431596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.429032    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea678e2-cfc3-4ff5-8889-601be2dd2909-istiod-ca-cert\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.431596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.429077    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zjm\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-kube-api-access-25zjm\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.431596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.429195    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-data\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.431596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.429247    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-socket\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.431596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.429746    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-socket\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.431596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.430283    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-credential-socket\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.469167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.433661    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea678e2-cfc3-4ff5-8889-601be2dd2909-istiod-ca-cert\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.469167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.454701    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-data\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.469167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.458262    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zjm\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-kube-api-access-25zjm\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.469167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.464128    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-envoy\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.473596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.470022    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-token\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.478618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.476330    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-podinfo\") pod \"e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") " pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:12.776937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:12.775469    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:13.084914 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:13.084885    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea678e2_cfc3_4ff5_8889_601be2dd2909.slice/crio-417e801047465a701090828d11f649380be937d823aa56fe215526a0d68b3b76 WatchSource:0}: Error finding container 417e801047465a701090828d11f649380be937d823aa56fe215526a0d68b3b76: Status 404 returned error can't find the container with id 417e801047465a701090828d11f649380be937d823aa56fe215526a0d68b3b76
Nov 03 11:15:13.085789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.085765    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"]
Nov 03 11:15:13.088526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.088491    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:13.088678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.088584    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:13.088678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.088629    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:13.325596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.320207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:13.665356 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.665325    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" event={"ID":"dea678e2-cfc3-4ff5-8889-601be2dd2909","Type":"ContainerStarted","Data":"e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786"}
Nov 03 11:15:13.665522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.665387    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" event={"ID":"dea678e2-cfc3-4ff5-8889-601be2dd2909","Type":"ContainerStarted","Data":"417e801047465a701090828d11f649380be937d823aa56fe215526a0d68b3b76"}
Nov 03 11:15:13.685953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.685909    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podStartSLOduration=1.685894183 podStartE2EDuration="1.685894183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:13.683438904 +0000 UTC m=+2743.889020957" watchObservedRunningTime="2026-11-03 11:15:13.685894183 +0000 UTC m=+2743.891476235"
Nov 03 11:15:13.776118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:13.776092    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:14.263542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:14.263507    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:14.403059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:14.403028    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:14.541912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:14.541824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:14.782957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:14.782927    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:15.688528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:15.688495    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:15.689643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:15.689620    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:15:15.745679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:15.745199    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:15.897595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:15.897543    5817 scope.go:122] "RemoveContainer" containerID="34d5bb78a3eb4132130bcf307508e1eb3152b8de9238d9740d50b76ec58d17d0"
Nov 03 11:15:15.900348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:15.900057    5817 scope.go:122] "RemoveContainer" containerID="1f064800da004087287bef7c07bf782647701b3fdd1f5dd056de4f801c7297cd"
Nov 03 11:15:16.216162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.216078    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:16.400599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.388610    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:16.749140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.749110    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" event={"ID":"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6","Type":"ContainerStarted","Data":"70ba5009da9ae1fc07dac168e4695f98b5dec6bb189b54266121ef891c8a258e"}
Nov 03 11:15:16.894528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.894431    5817 scope.go:122] "RemoveContainer" containerID="b8a932f31e3d50ff3723bfad47c16ce0f9da6ad746aa4a26961dbd4181d749e8"
Nov 03 11:15:16.894862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.894836    5817 scope.go:122] "RemoveContainer" containerID="091f04ecca72591f4a35adf003b4ac9197205e2f50d14e241d291fe1af15201b"
Nov 03 11:15:16.895079 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.895062    5817 scope.go:122] "RemoveContainer" containerID="ce66cd30365b159f84c15eb53a330d4bd5fea6f803ac22066d942ba7d15c52fa"
Nov 03 11:15:16.895193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.895175    5817 scope.go:122] "RemoveContainer" containerID="162dbd6044746dc8911734263add382fc3ef556dce4d381b946e25d4c2d57bea"
Nov 03 11:15:16.895360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.895312    5817 scope.go:122] "RemoveContainer" containerID="c669e591d6f743cfd32489da19c65980f911c5cf5f2c29c23bc3853e1522d3ce"
Nov 03 11:15:16.895492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:16.895473    5817 scope.go:122] "RemoveContainer" containerID="1e49ca6e8f3b20855382a02523bef8bf86a27d026d95e4d1bd033461aee2e22b"
Nov 03 11:15:17.504551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.504521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:17.799560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.797282    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:17.879596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.869156    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"]
Nov 03 11:15:17.889605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.882545    5817 generic.go:356] "Generic (PLEG): container finished" podID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerID="2aa7c940ccdccad0b4320ecd8e410162ca5eface91e4ed6e0c1334ad92661c8f" exitCode=60
Nov 03 11:15:17.890193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.889785    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" event={"ID":"24da8686-666d-4c92-81fa-5e6a07e4c5bb","Type":"ContainerDied","Data":"2aa7c940ccdccad0b4320ecd8e410162ca5eface91e4ed6e0c1334ad92661c8f"}
Nov 03 11:15:17.890389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.890378    5817 scope.go:122] "RemoveContainer" containerID="34d5bb78a3eb4132130bcf307508e1eb3152b8de9238d9740d50b76ec58d17d0"
Nov 03 11:15:17.891446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.891429    5817 scope.go:122] "RemoveContainer" containerID="2aa7c940ccdccad0b4320ecd8e410162ca5eface91e4ed6e0c1334ad92661c8f"
Nov 03 11:15:17.892097 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:17.892072    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss_odh-ai-gateway-infra(24da8686-666d-4c92-81fa-5e6a07e4c5bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb"
Nov 03 11:15:17.925776 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.925748    5817 scope.go:122] "RemoveContainer" containerID="9ebc224dd1e01dd02211bdfd6968714e3cf139ecd36e14d743da6085fcc8717b"
Nov 03 11:15:17.939619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.927241    5817 scope.go:122] "RemoveContainer" containerID="537e0537e0b4f0245c0d8baaacde49027b04d85db0b93642674960b94cce35e4"
Nov 03 11:15:17.943432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.940227    5817 scope.go:122] "RemoveContainer" containerID="b68e28287ad3c5c849428532de64b74810b1de822a817b65664486db8423be34"
Nov 03 11:15:17.943432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:17.940896    5817 scope.go:122] "RemoveContainer" containerID="d55f8648d464735f54e7c369a9548f10a9ab94222c43d988fe31c7c238f0b769"
Nov 03 11:15:18.077857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.077818    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" event={"ID":"6da14868-435e-4ed1-96a3-b295fa1ac1a1","Type":"ContainerStarted","Data":"075e5a1dd917a70ca3826de2dd2ba27c81a46a6ed430ee2fb0f61479bfbfc075"}
Nov 03 11:15:18.148219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.147977    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" event={"ID":"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464","Type":"ContainerStarted","Data":"84fba794e5310227512e83e785d802ad8f6e58c273e817e10faa7e58d0c0a7a1"}
Nov 03 11:15:18.152322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.152284    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:18.224599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.223287    5817 generic.go:356] "Generic (PLEG): container finished" podID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerID="70ba5009da9ae1fc07dac168e4695f98b5dec6bb189b54266121ef891c8a258e" exitCode=60
Nov 03 11:15:18.224599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.223533    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" event={"ID":"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6","Type":"ContainerDied","Data":"70ba5009da9ae1fc07dac168e4695f98b5dec6bb189b54266121ef891c8a258e"}
Nov 03 11:15:18.276532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.276507    5817 scope.go:122] "RemoveContainer" containerID="1f064800da004087287bef7c07bf782647701b3fdd1f5dd056de4f801c7297cd"
Nov 03 11:15:18.357723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.356976    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:18.560953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.560896    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:18.605029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.604973    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-kube-api-access-xqfph\" (UniqueName: \"kubernetes.io/projected/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-kube-api-access-xqfph\") pod \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\" (UID: \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\") "
Nov 03 11:15:18.605735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.605317    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-maas-api-service-ca\") pod \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\" (UID: \"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6\") "
Nov 03 11:15:18.607302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.606870    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-maas-api-service-ca" pod "55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" (UID: "55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:18.611094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.611072    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-kube-api-access-xqfph" pod "55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" (UID: "55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6"). InnerVolumeSpecName "kube-api-access-xqfph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:18.708420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.708392    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfph\" (UniqueName: \"kubernetes.io/projected/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-kube-api-access-xqfph\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:18.708420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.708422    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:18.896379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.896353    5817 scope.go:122] "RemoveContainer" containerID="40acc8fa1797b6ad6bdad7dba266d1fcde7dcd649a073330cc7308dfce85e1b2"
Nov 03 11:15:18.896993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.896972    5817 scope.go:122] "RemoveContainer" containerID="d68a3b1f7553b4bb15742f6d21b77564f9223e67f96760d1ffd37c748d0498d9"
Nov 03 11:15:18.928020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:18.927989    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"]
Nov 03 11:15:19.163696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.159298    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"]
Nov 03 11:15:19.172984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.171915    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:19.245345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.245310    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"]
Nov 03 11:15:19.270929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.270898    5817 generic.go:356] "Generic (PLEG): container finished" podID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerID="7a2a140557590c423521a817976c000b496d442bdf8746955cbe74e997ad2854" exitCode=60
Nov 03 11:15:19.271119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.270968    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv" event={"ID":"38af2eca-791e-4da1-a74d-f93b4a6fa8e9","Type":"ContainerDied","Data":"7a2a140557590c423521a817976c000b496d442bdf8746955cbe74e997ad2854"}
Nov 03 11:15:19.271119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.271029    5817 scope.go:122] "RemoveContainer" containerID="1e49ca6e8f3b20855382a02523bef8bf86a27d026d95e4d1bd033461aee2e22b"
Nov 03 11:15:19.271523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.271498    5817 scope.go:122] "RemoveContainer" containerID="7a2a140557590c423521a817976c000b496d442bdf8746955cbe74e997ad2854"
Nov 03 11:15:19.272075 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:19.272047    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv_odh-ai-gateway-infra(38af2eca-791e-4da1-a74d-f93b4a6fa8e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9"
Nov 03 11:15:19.275220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.275191    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" event={"ID":"f0967a8b-43f1-496a-8a5c-03e435f50232","Type":"ContainerStarted","Data":"2f728ba6adf1a34487556156316b08a0d618a72723e51372343c95cce801a30c"}
Nov 03 11:15:19.278394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.278295    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" event={"ID":"63a0333b-9ed5-4e5a-95cf-ea734272085b","Type":"ContainerStarted","Data":"8508833c69a1a4ec3577c523ac31131c86c08822f5c7d56cd8d6dd2ec9a7a869"}
Nov 03 11:15:19.280651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.280623    5817 generic.go:356] "Generic (PLEG): container finished" podID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerID="923e7f27fbba8e8c11cb1fc55e30d8b6f033675e5972b6c69fa44057534b5c3a" exitCode=60
Nov 03 11:15:19.280762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.280744    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" event={"ID":"8f21b3aa-337d-46be-ae7a-c5658f48b0af","Type":"ContainerDied","Data":"923e7f27fbba8e8c11cb1fc55e30d8b6f033675e5972b6c69fa44057534b5c3a"}
Nov 03 11:15:19.281165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.281139    5817 scope.go:122] "RemoveContainer" containerID="923e7f27fbba8e8c11cb1fc55e30d8b6f033675e5972b6c69fa44057534b5c3a"
Nov 03 11:15:19.281671 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:19.281643    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc_odh-ai-gateway-infra(8f21b3aa-337d-46be-ae7a-c5658f48b0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af"
Nov 03 11:15:19.283302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.283287    5817 generic.go:356] "Generic (PLEG): container finished" podID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerID="ca2fcb69c11c0e3d12aa9cbf862b44f8d5b65227b08881a7b5a152dab6b3f10a" exitCode=60
Nov 03 11:15:19.283438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.283425    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" event={"ID":"90118aeb-a009-4ce0-b6dd-81c1e08b26e9","Type":"ContainerDied","Data":"ca2fcb69c11c0e3d12aa9cbf862b44f8d5b65227b08881a7b5a152dab6b3f10a"}
Nov 03 11:15:19.283866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.283853    5817 scope.go:122] "RemoveContainer" containerID="ca2fcb69c11c0e3d12aa9cbf862b44f8d5b65227b08881a7b5a152dab6b3f10a"
Nov 03 11:15:19.284178 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:19.284161    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp_odh-ai-gateway-infra(90118aeb-a009-4ce0-b6dd-81c1e08b26e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9"
Nov 03 11:15:19.293908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.293878    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" event={"ID":"a386c604-04c2-4d21-bd0d-99b4bc965c3e","Type":"ContainerStarted","Data":"2dd202591a4992b34a7105007da40a2f55f0f5ff10e189fbd0ca3db9960af5ac"}
Nov 03 11:15:19.299231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.299213    5817 scope.go:122] "RemoveContainer" containerID="091f04ecca72591f4a35adf003b4ac9197205e2f50d14e241d291fe1af15201b"
Nov 03 11:15:19.300096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.300074    5817 generic.go:356] "Generic (PLEG): container finished" podID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerID="075e5a1dd917a70ca3826de2dd2ba27c81a46a6ed430ee2fb0f61479bfbfc075" exitCode=60
Nov 03 11:15:19.301170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.300140    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" event={"ID":"6da14868-435e-4ed1-96a3-b295fa1ac1a1","Type":"ContainerDied","Data":"075e5a1dd917a70ca3826de2dd2ba27c81a46a6ed430ee2fb0f61479bfbfc075"}
Nov 03 11:15:19.303385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.303364    5817 generic.go:356] "Generic (PLEG): container finished" podID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerID="84fba794e5310227512e83e785d802ad8f6e58c273e817e10faa7e58d0c0a7a1" exitCode=60
Nov 03 11:15:19.303484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.303418    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" event={"ID":"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464","Type":"ContainerDied","Data":"84fba794e5310227512e83e785d802ad8f6e58c273e817e10faa7e58d0c0a7a1"}
Nov 03 11:15:19.306759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.306740    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" event={"ID":"fbc01256-4c3b-40ce-9b18-7b20596c7255","Type":"ContainerStarted","Data":"e0c35f76d6f22cc3879baddaf4659ede09239d13417b92190c5fb4ae6f27d384"}
Nov 03 11:15:19.309310 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.309287    5817 generic.go:356] "Generic (PLEG): container finished" podID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerID="7a42ccf3e9851d2725d1d106dd20d3721f05ce5d9cf43982140f3dd6392a7285" exitCode=60
Nov 03 11:15:19.309402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.309359    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr" event={"ID":"154bc19a-4ea4-47ea-9a6a-d5c3920f902d","Type":"ContainerDied","Data":"7a42ccf3e9851d2725d1d106dd20d3721f05ce5d9cf43982140f3dd6392a7285"}
Nov 03 11:15:19.309697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.309677    5817 scope.go:122] "RemoveContainer" containerID="7a42ccf3e9851d2725d1d106dd20d3721f05ce5d9cf43982140f3dd6392a7285"
Nov 03 11:15:19.310037 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:19.310013    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr_odh-ai-gateway-infra(154bc19a-4ea4-47ea-9a6a-d5c3920f902d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d"
Nov 03 11:15:19.311477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.311461    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"
Nov 03 11:15:19.311622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.311508    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5" event={"ID":"55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6","Type":"ContainerDied","Data":"2cdb00bbdf5cc91b9078fafcb7613f2f9e817fe167b73267ffe2bc5e4d854b3a"}
Nov 03 11:15:19.322133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.322114    5817 scope.go:122] "RemoveContainer" containerID="b8a932f31e3d50ff3723bfad47c16ce0f9da6ad746aa4a26961dbd4181d749e8"
Nov 03 11:15:19.341470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.341445    5817 scope.go:122] "RemoveContainer" containerID="ce66cd30365b159f84c15eb53a330d4bd5fea6f803ac22066d942ba7d15c52fa"
Nov 03 11:15:19.360011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.359989    5817 scope.go:122] "RemoveContainer" containerID="162dbd6044746dc8911734263add382fc3ef556dce4d381b946e25d4c2d57bea"
Nov 03 11:15:19.376022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.376002    5817 scope.go:122] "RemoveContainer" containerID="c669e591d6f743cfd32489da19c65980f911c5cf5f2c29c23bc3853e1522d3ce"
Nov 03 11:15:19.395214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.395097    5817 scope.go:122] "RemoveContainer" containerID="70ba5009da9ae1fc07dac168e4695f98b5dec6bb189b54266121ef891c8a258e"
Nov 03 11:15:19.634498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.634443    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"]
Nov 03 11:15:19.649422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.649391    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895075-wwjg5"]
Nov 03 11:15:19.863292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.863234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:19.958094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.955497    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" path="/var/lib/kubelet/pods/55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6/volumes"
Nov 03 11:15:19.975317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.975283    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:19.977420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.977397    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:19.979438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:19.979409    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:20.104391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.104354    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-maas-api-service-ca\") pod \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\" (UID: \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\") "
Nov 03 11:15:20.104594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.104425    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da14868-435e-4ed1-96a3-b295fa1ac1a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14868-435e-4ed1-96a3-b295fa1ac1a1-maas-api-service-ca\") pod \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\" (UID: \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\") "
Nov 03 11:15:20.104594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.104509    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da14868-435e-4ed1-96a3-b295fa1ac1a1-kube-api-access-skg26\" (UniqueName: \"kubernetes.io/projected/6da14868-435e-4ed1-96a3-b295fa1ac1a1-kube-api-access-skg26\") pod \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\" (UID: \"6da14868-435e-4ed1-96a3-b295fa1ac1a1\") "
Nov 03 11:15:20.104722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.104616    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24da8686-666d-4c92-81fa-5e6a07e4c5bb-kube-api-access-wnwx7\" (UniqueName: \"kubernetes.io/projected/24da8686-666d-4c92-81fa-5e6a07e4c5bb-kube-api-access-wnwx7\") pod \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\" (UID: \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\") "
Nov 03 11:15:20.104782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.104732    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24da8686-666d-4c92-81fa-5e6a07e4c5bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24da8686-666d-4c92-81fa-5e6a07e4c5bb-maas-api-service-ca\") pod \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\" (UID: \"24da8686-666d-4c92-81fa-5e6a07e4c5bb\") "
Nov 03 11:15:20.104782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.104764    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-kube-api-access-cq2rl\" (UniqueName: \"kubernetes.io/projected/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-kube-api-access-cq2rl\") pod \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\" (UID: \"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464\") "
Nov 03 11:15:20.104906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.104824    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da14868-435e-4ed1-96a3-b295fa1ac1a1-maas-api-service-ca" pod "6da14868-435e-4ed1-96a3-b295fa1ac1a1" (UID: "6da14868-435e-4ed1-96a3-b295fa1ac1a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:20.105488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.105465    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-maas-api-service-ca" pod "72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" (UID: "72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:20.105868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.105848    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.105970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.105873    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14868-435e-4ed1-96a3-b295fa1ac1a1-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.106191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.106173    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24da8686-666d-4c92-81fa-5e6a07e4c5bb-maas-api-service-ca" pod "24da8686-666d-4c92-81fa-5e6a07e4c5bb" (UID: "24da8686-666d-4c92-81fa-5e6a07e4c5bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:20.108587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.108553    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da14868-435e-4ed1-96a3-b295fa1ac1a1-kube-api-access-skg26" pod "6da14868-435e-4ed1-96a3-b295fa1ac1a1" (UID: "6da14868-435e-4ed1-96a3-b295fa1ac1a1"). InnerVolumeSpecName "kube-api-access-skg26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:20.109350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.109329    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24da8686-666d-4c92-81fa-5e6a07e4c5bb-kube-api-access-wnwx7" pod "24da8686-666d-4c92-81fa-5e6a07e4c5bb" (UID: "24da8686-666d-4c92-81fa-5e6a07e4c5bb"). InnerVolumeSpecName "kube-api-access-wnwx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:20.110463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.110433    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-kube-api-access-cq2rl" pod "72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" (UID: "72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464"). InnerVolumeSpecName "kube-api-access-cq2rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:20.207832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.207795    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skg26\" (UniqueName: \"kubernetes.io/projected/6da14868-435e-4ed1-96a3-b295fa1ac1a1-kube-api-access-skg26\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.208054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.207836    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnwx7\" (UniqueName: \"kubernetes.io/projected/24da8686-666d-4c92-81fa-5e6a07e4c5bb-kube-api-access-wnwx7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.208054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.207854    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24da8686-666d-4c92-81fa-5e6a07e4c5bb-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.208054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.207869    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq2rl\" (UniqueName: \"kubernetes.io/projected/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464-kube-api-access-cq2rl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.302409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.302379    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"]
Nov 03 11:15:20.330829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.330801    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"]
Nov 03 11:15:20.337756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.337729    5817 generic.go:356] "Generic (PLEG): container finished" podID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerID="e0c35f76d6f22cc3879baddaf4659ede09239d13417b92190c5fb4ae6f27d384" exitCode=60
Nov 03 11:15:20.337919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.337804    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" event={"ID":"fbc01256-4c3b-40ce-9b18-7b20596c7255","Type":"ContainerDied","Data":"e0c35f76d6f22cc3879baddaf4659ede09239d13417b92190c5fb4ae6f27d384"}
Nov 03 11:15:20.337919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.337860    5817 scope.go:122] "RemoveContainer" containerID="d55f8648d464735f54e7c369a9548f10a9ab94222c43d988fe31c7c238f0b769"
Nov 03 11:15:20.338280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.338252    5817 scope.go:122] "RemoveContainer" containerID="e0c35f76d6f22cc3879baddaf4659ede09239d13417b92190c5fb4ae6f27d384"
Nov 03 11:15:20.338766 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:20.338723    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t_odh-ai-gateway-infra(fbc01256-4c3b-40ce-9b18-7b20596c7255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255"
Nov 03 11:15:20.346801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.346774    5817 generic.go:356] "Generic (PLEG): container finished" podID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerID="2f728ba6adf1a34487556156316b08a0d618a72723e51372343c95cce801a30c" exitCode=60
Nov 03 11:15:20.346907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.346845    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" event={"ID":"f0967a8b-43f1-496a-8a5c-03e435f50232","Type":"ContainerDied","Data":"2f728ba6adf1a34487556156316b08a0d618a72723e51372343c95cce801a30c"}
Nov 03 11:15:20.347327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.347262    5817 scope.go:122] "RemoveContainer" containerID="2f728ba6adf1a34487556156316b08a0d618a72723e51372343c95cce801a30c"
Nov 03 11:15:20.347714 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:20.347690    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc_odh-ai-gateway-infra(f0967a8b-43f1-496a-8a5c-03e435f50232)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232"
Nov 03 11:15:20.349766 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.349747    5817 generic.go:356] "Generic (PLEG): container finished" podID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerID="8508833c69a1a4ec3577c523ac31131c86c08822f5c7d56cd8d6dd2ec9a7a869" exitCode=60
Nov 03 11:15:20.349950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.349931    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" event={"ID":"63a0333b-9ed5-4e5a-95cf-ea734272085b","Type":"ContainerDied","Data":"8508833c69a1a4ec3577c523ac31131c86c08822f5c7d56cd8d6dd2ec9a7a869"}
Nov 03 11:15:20.350760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.350526    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"]
Nov 03 11:15:20.357874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.357854    5817 scope.go:122] "RemoveContainer" containerID="9ebc224dd1e01dd02211bdfd6968714e3cf139ecd36e14d743da6085fcc8717b"
Nov 03 11:15:20.362950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.362014    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"
Nov 03 11:15:20.362950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.362414    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss" event={"ID":"24da8686-666d-4c92-81fa-5e6a07e4c5bb","Type":"ContainerDied","Data":"c71e6fc29067fa63015df4c070c2e8fe34f998677bb13d4d6c8b16028bd93b17"}
Nov 03 11:15:20.366341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.366206    5817 generic.go:356] "Generic (PLEG): container finished" podID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerID="42eb52716df1369ab3ca8bf6ee42cf8b00d2564ea8deabbe1f70123a9f5dbe5e" exitCode=60
Nov 03 11:15:20.366341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.366315    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" event={"ID":"d9a126cd-c401-485f-90f9-eaeb57262e03","Type":"ContainerDied","Data":"42eb52716df1369ab3ca8bf6ee42cf8b00d2564ea8deabbe1f70123a9f5dbe5e"}
Nov 03 11:15:20.366961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.366939    5817 scope.go:122] "RemoveContainer" containerID="42eb52716df1369ab3ca8bf6ee42cf8b00d2564ea8deabbe1f70123a9f5dbe5e"
Nov 03 11:15:20.367648 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:20.367357    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl_odh-ai-gateway-infra(d9a126cd-c401-485f-90f9-eaeb57262e03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03"
Nov 03 11:15:20.370315 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.370290    5817 generic.go:356] "Generic (PLEG): container finished" podID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerID="2dd202591a4992b34a7105007da40a2f55f0f5ff10e189fbd0ca3db9960af5ac" exitCode=60
Nov 03 11:15:20.370410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.370381    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" event={"ID":"a386c604-04c2-4d21-bd0d-99b4bc965c3e","Type":"ContainerDied","Data":"2dd202591a4992b34a7105007da40a2f55f0f5ff10e189fbd0ca3db9960af5ac"}
Nov 03 11:15:20.370955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.370931    5817 scope.go:122] "RemoveContainer" containerID="2dd202591a4992b34a7105007da40a2f55f0f5ff10e189fbd0ca3db9960af5ac"
Nov 03 11:15:20.371341 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:20.371316    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs_odh-ai-gateway-infra(a386c604-04c2-4d21-bd0d-99b4bc965c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e"
Nov 03 11:15:20.375002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.374975    5817 generic.go:356] "Generic (PLEG): container finished" podID="2d0566a2-4971-4901-a926-af80b11a0603" containerID="0665d01dcf9e189b1751be5ec79f68db41cb04b1988563de4d7e4fd813b37f95" exitCode=60
Nov 03 11:15:20.375121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.375065    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" event={"ID":"2d0566a2-4971-4901-a926-af80b11a0603","Type":"ContainerDied","Data":"0665d01dcf9e189b1751be5ec79f68db41cb04b1988563de4d7e4fd813b37f95"}
Nov 03 11:15:20.375504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.375488    5817 scope.go:122] "RemoveContainer" containerID="0665d01dcf9e189b1751be5ec79f68db41cb04b1988563de4d7e4fd813b37f95"
Nov 03 11:15:20.378429 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:20.377651    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd_odh-ai-gateway-infra(2d0566a2-4971-4901-a926-af80b11a0603)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" podUID="2d0566a2-4971-4901-a926-af80b11a0603"
Nov 03 11:15:20.378429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.377776    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"
Nov 03 11:15:20.378429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.377782    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk" event={"ID":"6da14868-435e-4ed1-96a3-b295fa1ac1a1","Type":"ContainerDied","Data":"bfb604d0a4e5de4fefcf9c12967ef9ca4e075eec76189ea16fbc22d966a10cae"}
Nov 03 11:15:20.383111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.383085    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr" event={"ID":"72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464","Type":"ContainerDied","Data":"7e3ac78b71d4c75152df2b1642230313638c6b507efc1409fa4bccd31245981b"}
Nov 03 11:15:20.383221 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.383204    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"
Nov 03 11:15:20.385596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.385539    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"]
Nov 03 11:15:20.397716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.397692    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"]
Nov 03 11:15:20.424915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.424546    5817 scope.go:122] "RemoveContainer" containerID="537e0537e0b4f0245c0d8baaacde49027b04d85db0b93642674960b94cce35e4"
Nov 03 11:15:20.460314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.460237    5817 scope.go:122] "RemoveContainer" containerID="2aa7c940ccdccad0b4320ecd8e410162ca5eface91e4ed6e0c1334ad92661c8f"
Nov 03 11:15:20.484067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.482595    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"]
Nov 03 11:15:20.498462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.498436    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"]
Nov 03 11:15:20.565617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.563756    5817 scope.go:122] "RemoveContainer" containerID="d68a3b1f7553b4bb15742f6d21b77564f9223e67f96760d1ffd37c748d0498d9"
Nov 03 11:15:20.572722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.567411    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:20.662166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.662138    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"]
Nov 03 11:15:20.673190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.673160    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895075-tsnfk"]
Nov 03 11:15:20.685942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.685849    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"]
Nov 03 11:15:20.723240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.722864    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:20.732801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.732356    5817 scope.go:122] "RemoveContainer" containerID="b68e28287ad3c5c849428532de64b74810b1de822a817b65664486db8423be34"
Nov 03 11:15:20.742996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.742948    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-kube-api-access-2qhzl\" (UniqueName: \"kubernetes.io/projected/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-kube-api-access-2qhzl\") pod \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\" (UID: \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\") "
Nov 03 11:15:20.744341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.743335    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-maas-api-service-ca\") pod \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\" (UID: \"38af2eca-791e-4da1-a74d-f93b4a6fa8e9\") "
Nov 03 11:15:20.744957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.744649    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-maas-api-service-ca" pod "38af2eca-791e-4da1-a74d-f93b4a6fa8e9" (UID: "38af2eca-791e-4da1-a74d-f93b4a6fa8e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:20.746533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.746516    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"]
Nov 03 11:15:20.749241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.749201    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-kube-api-access-2qhzl" pod "38af2eca-791e-4da1-a74d-f93b4a6fa8e9" (UID: "38af2eca-791e-4da1-a74d-f93b4a6fa8e9"). InnerVolumeSpecName "kube-api-access-2qhzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:20.759361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.759338    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d9f2ce-29895075-25kss"]
Nov 03 11:15:20.772654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.772629    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"]
Nov 03 11:15:20.782646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.782585    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895075-crhfr"]
Nov 03 11:15:20.850878 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.850307    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qhzl\" (UniqueName: \"kubernetes.io/projected/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-kube-api-access-2qhzl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.850878 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.850354    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af2eca-791e-4da1-a74d-f93b4a6fa8e9-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.875766 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.875743    5817 scope.go:122] "RemoveContainer" containerID="40acc8fa1797b6ad6bdad7dba266d1fcde7dcd649a073330cc7308dfce85e1b2"
Nov 03 11:15:20.900001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.899965    5817 scope.go:122] "RemoveContainer" containerID="0e5e1dc6aece02b659dc54fcb403ff720e01d6bcc5d5736f53bf7cdb3f7d9279"
Nov 03 11:15:20.908843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.907327    5817 scope.go:122] "RemoveContainer" containerID="075e5a1dd917a70ca3826de2dd2ba27c81a46a6ed430ee2fb0f61479bfbfc075"
Nov 03 11:15:20.942757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:20.942513    5817 scope.go:122] "RemoveContainer" containerID="84fba794e5310227512e83e785d802ad8f6e58c273e817e10faa7e58d0c0a7a1"
Nov 03 11:15:21.013525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.013037    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:21.017561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.016923    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:21.041118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.041042    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:21.164323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.164291    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a0333b-9ed5-4e5a-95cf-ea734272085b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a0333b-9ed5-4e5a-95cf-ea734272085b-maas-api-service-ca\") pod \"63a0333b-9ed5-4e5a-95cf-ea734272085b\" (UID: \"63a0333b-9ed5-4e5a-95cf-ea734272085b\") "
Nov 03 11:15:21.164696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.164675    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-maas-api-service-ca\") pod \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\" (UID: \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\") "
Nov 03 11:15:21.164927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.164912    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-kube-api-access-xnjjv\" (UniqueName: \"kubernetes.io/projected/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-kube-api-access-xnjjv\") pod \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\" (UID: \"90118aeb-a009-4ce0-b6dd-81c1e08b26e9\") "
Nov 03 11:15:21.165056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.165042    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f21b3aa-337d-46be-ae7a-c5658f48b0af-kube-api-access-6hppc\" (UniqueName: \"kubernetes.io/projected/8f21b3aa-337d-46be-ae7a-c5658f48b0af-kube-api-access-6hppc\") pod \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\" (UID: \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\") "
Nov 03 11:15:21.165190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.165176    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a0333b-9ed5-4e5a-95cf-ea734272085b-kube-api-access-vn96v\" (UniqueName: \"kubernetes.io/projected/63a0333b-9ed5-4e5a-95cf-ea734272085b-kube-api-access-vn96v\") pod \"63a0333b-9ed5-4e5a-95cf-ea734272085b\" (UID: \"63a0333b-9ed5-4e5a-95cf-ea734272085b\") "
Nov 03 11:15:21.165388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.165375    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f21b3aa-337d-46be-ae7a-c5658f48b0af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21b3aa-337d-46be-ae7a-c5658f48b0af-maas-api-service-ca\") pod \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\" (UID: \"8f21b3aa-337d-46be-ae7a-c5658f48b0af\") "
Nov 03 11:15:21.166831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.166781    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f21b3aa-337d-46be-ae7a-c5658f48b0af-maas-api-service-ca" pod "8f21b3aa-337d-46be-ae7a-c5658f48b0af" (UID: "8f21b3aa-337d-46be-ae7a-c5658f48b0af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:21.167155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.167132    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a0333b-9ed5-4e5a-95cf-ea734272085b-maas-api-service-ca" pod "63a0333b-9ed5-4e5a-95cf-ea734272085b" (UID: "63a0333b-9ed5-4e5a-95cf-ea734272085b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:21.168085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.168000    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-maas-api-service-ca" pod "90118aeb-a009-4ce0-b6dd-81c1e08b26e9" (UID: "90118aeb-a009-4ce0-b6dd-81c1e08b26e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:21.192270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.192220    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f21b3aa-337d-46be-ae7a-c5658f48b0af-kube-api-access-6hppc" pod "8f21b3aa-337d-46be-ae7a-c5658f48b0af" (UID: "8f21b3aa-337d-46be-ae7a-c5658f48b0af"). InnerVolumeSpecName "kube-api-access-6hppc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:21.195175 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.195057    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a0333b-9ed5-4e5a-95cf-ea734272085b-kube-api-access-vn96v" pod "63a0333b-9ed5-4e5a-95cf-ea734272085b" (UID: "63a0333b-9ed5-4e5a-95cf-ea734272085b"). InnerVolumeSpecName "kube-api-access-vn96v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:21.195947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.195921    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-kube-api-access-xnjjv" pod "90118aeb-a009-4ce0-b6dd-81c1e08b26e9" (UID: "90118aeb-a009-4ce0-b6dd-81c1e08b26e9"). InnerVolumeSpecName "kube-api-access-xnjjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:21.262890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.262821    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:21.267677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.267653    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn96v\" (UniqueName: \"kubernetes.io/projected/63a0333b-9ed5-4e5a-95cf-ea734272085b-kube-api-access-vn96v\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.267677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.267678    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21b3aa-337d-46be-ae7a-c5658f48b0af-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.267879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.267694    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a0333b-9ed5-4e5a-95cf-ea734272085b-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.267879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.267708    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.267879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.267720    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnjjv\" (UniqueName: \"kubernetes.io/projected/90118aeb-a009-4ce0-b6dd-81c1e08b26e9-kube-api-access-xnjjv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.267879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.267734    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hppc\" (UniqueName: \"kubernetes.io/projected/8f21b3aa-337d-46be-ae7a-c5658f48b0af-kube-api-access-6hppc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.368796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.368764    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-maas-api-service-ca\") pod \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\" (UID: \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\") "
Nov 03 11:15:21.368967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.368834    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-kube-api-access-gd5kn\") pod \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\" (UID: \"154bc19a-4ea4-47ea-9a6a-d5c3920f902d\") "
Nov 03 11:15:21.369168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.369141    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-maas-api-service-ca" pod "154bc19a-4ea4-47ea-9a6a-d5c3920f902d" (UID: "154bc19a-4ea4-47ea-9a6a-d5c3920f902d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:21.369614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.369597    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.371170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.371145    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-kube-api-access-gd5kn" pod "154bc19a-4ea4-47ea-9a6a-d5c3920f902d" (UID: "154bc19a-4ea4-47ea-9a6a-d5c3920f902d"). InnerVolumeSpecName "kube-api-access-gd5kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:21.424611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.416214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:21.465864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.465805    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr" event={"ID":"154bc19a-4ea4-47ea-9a6a-d5c3920f902d","Type":"ContainerDied","Data":"76f58cf0999f0d9a0f70f92b7905dfe886ab2552dee05df372f5adccaf4ac178"}
Nov 03 11:15:21.466191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.466168    5817 scope.go:122] "RemoveContainer" containerID="7a42ccf3e9851d2725d1d106dd20d3721f05ce5d9cf43982140f3dd6392a7285"
Nov 03 11:15:21.466515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.466499    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"
Nov 03 11:15:21.475306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.475152    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/154bc19a-4ea4-47ea-9a6a-d5c3920f902d-kube-api-access-gd5kn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.505960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.492565    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv" event={"ID":"38af2eca-791e-4da1-a74d-f93b4a6fa8e9","Type":"ContainerDied","Data":"32f30d412a29021669c408e6b793726ef319b4cf09ddf87822481e697954a9df"}
Nov 03 11:15:21.505960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.493211    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"
Nov 03 11:15:21.521650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.519390    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c" event={"ID":"63a0333b-9ed5-4e5a-95cf-ea734272085b","Type":"ContainerDied","Data":"14cf12ee29ef4734ec75af7e0f32ac70cc08029e1137805600975a44e66821fb"}
Nov 03 11:15:21.521650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.519565    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"
Nov 03 11:15:21.533644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.529766    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc" event={"ID":"8f21b3aa-337d-46be-ae7a-c5658f48b0af","Type":"ContainerDied","Data":"722b30aa8c7eccb9875ac2750faaad2bf930b3f5745a2c3f5d7b0646a6355c6c"}
Nov 03 11:15:21.533644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.529895    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"
Nov 03 11:15:21.538592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.535638    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp" event={"ID":"90118aeb-a009-4ce0-b6dd-81c1e08b26e9","Type":"ContainerDied","Data":"ad8eebf3b0959f6fdf1ee4d78f60aa570d3ca5928d968ceb9c8aafa4fa0f79a0"}
Nov 03 11:15:21.538592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.535752    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"
Nov 03 11:15:21.549520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.549500    5817 scope.go:122] "RemoveContainer" containerID="7a2a140557590c423521a817976c000b496d442bdf8746955cbe74e997ad2854"
Nov 03 11:15:21.556697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.556503    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"]
Nov 03 11:15:21.610160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.610134    5817 scope.go:122] "RemoveContainer" containerID="8508833c69a1a4ec3577c523ac31131c86c08822f5c7d56cd8d6dd2ec9a7a869"
Nov 03 11:15:21.636832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.636736    5817 scope.go:122] "RemoveContainer" containerID="923e7f27fbba8e8c11cb1fc55e30d8b6f033675e5972b6c69fa44057534b5c3a"
Nov 03 11:15:21.638044 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.637816    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"]
Nov 03 11:15:21.683784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.683646    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"]
Nov 03 11:15:21.683784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.683701    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895075-b5c2c"]
Nov 03 11:15:21.690796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.690768    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"]
Nov 03 11:15:21.768125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.767961    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895075-2g9lv"]
Nov 03 11:15:21.785919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.785880    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"]
Nov 03 11:15:21.796351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.796324    5817 scope.go:122] "RemoveContainer" containerID="ca2fcb69c11c0e3d12aa9cbf862b44f8d5b65227b08881a7b5a152dab6b3f10a"
Nov 03 11:15:21.813349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.813268    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895075-n7jjp"]
Nov 03 11:15:21.885078 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.885052    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"]
Nov 03 11:15:21.926805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.923073    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" path="/var/lib/kubelet/pods/24da8686-666d-4c92-81fa-5e6a07e4c5bb/volumes"
Nov 03 11:15:21.926805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.923649    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" path="/var/lib/kubelet/pods/38af2eca-791e-4da1-a74d-f93b4a6fa8e9/volumes"
Nov 03 11:15:21.926805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.924250    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" path="/var/lib/kubelet/pods/63a0333b-9ed5-4e5a-95cf-ea734272085b/volumes"
Nov 03 11:15:21.926805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.924986    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" path="/var/lib/kubelet/pods/6da14868-435e-4ed1-96a3-b295fa1ac1a1/volumes"
Nov 03 11:15:21.926805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.925399    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" path="/var/lib/kubelet/pods/72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464/volumes"
Nov 03 11:15:21.926805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.925901    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" path="/var/lib/kubelet/pods/90118aeb-a009-4ce0-b6dd-81c1e08b26e9/volumes"
Nov 03 11:15:21.926805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.926531    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9f53bb-29895075-wfxwc"]
Nov 03 11:15:21.926805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.926559    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"]
Nov 03 11:15:21.932642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.932552    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895075-tjwtr"]
Nov 03 11:15:21.944713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:21.944328    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:22.003908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.003885    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:22.005261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.005239    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0967a8b-43f1-496a-8a5c-03e435f50232-kube-api-access-p9tlh\" (UniqueName: \"kubernetes.io/projected/f0967a8b-43f1-496a-8a5c-03e435f50232-kube-api-access-p9tlh\") pod \"f0967a8b-43f1-496a-8a5c-03e435f50232\" (UID: \"f0967a8b-43f1-496a-8a5c-03e435f50232\") "
Nov 03 11:15:22.006417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.006379    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0967a8b-43f1-496a-8a5c-03e435f50232-maas-api-service-ca" pod "f0967a8b-43f1-496a-8a5c-03e435f50232" (UID: "f0967a8b-43f1-496a-8a5c-03e435f50232"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.006551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.006015    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0967a8b-43f1-496a-8a5c-03e435f50232-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0967a8b-43f1-496a-8a5c-03e435f50232-maas-api-service-ca\") pod \"f0967a8b-43f1-496a-8a5c-03e435f50232\" (UID: \"f0967a8b-43f1-496a-8a5c-03e435f50232\") "
Nov 03 11:15:22.018508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.018449    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0967a8b-43f1-496a-8a5c-03e435f50232-kube-api-access-p9tlh" pod "f0967a8b-43f1-496a-8a5c-03e435f50232" (UID: "f0967a8b-43f1-496a-8a5c-03e435f50232"). InnerVolumeSpecName "kube-api-access-p9tlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.019342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.019318    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9tlh\" (UniqueName: \"kubernetes.io/projected/f0967a8b-43f1-496a-8a5c-03e435f50232-kube-api-access-p9tlh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.019474 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.019459    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0967a8b-43f1-496a-8a5c-03e435f50232-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.121385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.121354    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc01256-4c3b-40ce-9b18-7b20596c7255-kube-api-access-2vn84\" (UniqueName: \"kubernetes.io/projected/fbc01256-4c3b-40ce-9b18-7b20596c7255-kube-api-access-2vn84\") pod \"fbc01256-4c3b-40ce-9b18-7b20596c7255\" (UID: \"fbc01256-4c3b-40ce-9b18-7b20596c7255\") "
Nov 03 11:15:22.129461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.129428    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc01256-4c3b-40ce-9b18-7b20596c7255-kube-api-access-2vn84" pod "fbc01256-4c3b-40ce-9b18-7b20596c7255" (UID: "fbc01256-4c3b-40ce-9b18-7b20596c7255"). InnerVolumeSpecName "kube-api-access-2vn84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.129756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.129736    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc01256-4c3b-40ce-9b18-7b20596c7255-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc01256-4c3b-40ce-9b18-7b20596c7255-maas-api-service-ca\") pod \"fbc01256-4c3b-40ce-9b18-7b20596c7255\" (UID: \"fbc01256-4c3b-40ce-9b18-7b20596c7255\") "
Nov 03 11:15:22.130801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.130776    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vn84\" (UniqueName: \"kubernetes.io/projected/fbc01256-4c3b-40ce-9b18-7b20596c7255-kube-api-access-2vn84\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.130895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.130878    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc01256-4c3b-40ce-9b18-7b20596c7255-maas-api-service-ca" pod "fbc01256-4c3b-40ce-9b18-7b20596c7255" (UID: "fbc01256-4c3b-40ce-9b18-7b20596c7255"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.141654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.141298    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:22.143948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.143924    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:22.148889 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.148867    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:22.244152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.244114    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a386c604-04c2-4d21-bd0d-99b4bc965c3e-kube-api-access-tswvt\" (UniqueName: \"kubernetes.io/projected/a386c604-04c2-4d21-bd0d-99b4bc965c3e-kube-api-access-tswvt\") pod \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\" (UID: \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\") "
Nov 03 11:15:22.244427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.244401    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a386c604-04c2-4d21-bd0d-99b4bc965c3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386c604-04c2-4d21-bd0d-99b4bc965c3e-maas-api-service-ca\") pod \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\" (UID: \"a386c604-04c2-4d21-bd0d-99b4bc965c3e\") "
Nov 03 11:15:22.244504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.244487    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a126cd-c401-485f-90f9-eaeb57262e03-kube-api-access-zf5kp\" (UniqueName: \"kubernetes.io/projected/d9a126cd-c401-485f-90f9-eaeb57262e03-kube-api-access-zf5kp\") pod \"d9a126cd-c401-485f-90f9-eaeb57262e03\" (UID: \"d9a126cd-c401-485f-90f9-eaeb57262e03\") "
Nov 03 11:15:22.244594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.244563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0566a2-4971-4901-a926-af80b11a0603-kube-api-access-gsqdr\" (UniqueName: \"kubernetes.io/projected/2d0566a2-4971-4901-a926-af80b11a0603-kube-api-access-gsqdr\") pod \"2d0566a2-4971-4901-a926-af80b11a0603\" (UID: \"2d0566a2-4971-4901-a926-af80b11a0603\") "
Nov 03 11:15:22.244684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.244669    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0566a2-4971-4901-a926-af80b11a0603-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0566a2-4971-4901-a926-af80b11a0603-maas-api-service-ca\") pod \"2d0566a2-4971-4901-a926-af80b11a0603\" (UID: \"2d0566a2-4971-4901-a926-af80b11a0603\") "
Nov 03 11:15:22.244860 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.244843    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a126cd-c401-485f-90f9-eaeb57262e03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a126cd-c401-485f-90f9-eaeb57262e03-maas-api-service-ca\") pod \"d9a126cd-c401-485f-90f9-eaeb57262e03\" (UID: \"d9a126cd-c401-485f-90f9-eaeb57262e03\") "
Nov 03 11:15:22.245275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.245249    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0566a2-4971-4901-a926-af80b11a0603-maas-api-service-ca" pod "2d0566a2-4971-4901-a926-af80b11a0603" (UID: "2d0566a2-4971-4901-a926-af80b11a0603"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.245803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.245781    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc01256-4c3b-40ce-9b18-7b20596c7255-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.245901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.245807    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0566a2-4971-4901-a926-af80b11a0603-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.246082 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.246053    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a386c604-04c2-4d21-bd0d-99b4bc965c3e-maas-api-service-ca" pod "a386c604-04c2-4d21-bd0d-99b4bc965c3e" (UID: "a386c604-04c2-4d21-bd0d-99b4bc965c3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.246238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.246217    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a126cd-c401-485f-90f9-eaeb57262e03-maas-api-service-ca" pod "d9a126cd-c401-485f-90f9-eaeb57262e03" (UID: "d9a126cd-c401-485f-90f9-eaeb57262e03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.247988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.247950    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0566a2-4971-4901-a926-af80b11a0603-kube-api-access-gsqdr" pod "2d0566a2-4971-4901-a926-af80b11a0603" (UID: "2d0566a2-4971-4901-a926-af80b11a0603"). InnerVolumeSpecName "kube-api-access-gsqdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.248232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.248212    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a386c604-04c2-4d21-bd0d-99b4bc965c3e-kube-api-access-tswvt" pod "a386c604-04c2-4d21-bd0d-99b4bc965c3e" (UID: "a386c604-04c2-4d21-bd0d-99b4bc965c3e"). InnerVolumeSpecName "kube-api-access-tswvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.248678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.248645    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a126cd-c401-485f-90f9-eaeb57262e03-kube-api-access-zf5kp" pod "d9a126cd-c401-485f-90f9-eaeb57262e03" (UID: "d9a126cd-c401-485f-90f9-eaeb57262e03"). InnerVolumeSpecName "kube-api-access-zf5kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.394786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.384919    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:22.409031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.408672    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a126cd-c401-485f-90f9-eaeb57262e03-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.409285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.409265    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tswvt\" (UniqueName: \"kubernetes.io/projected/a386c604-04c2-4d21-bd0d-99b4bc965c3e-kube-api-access-tswvt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.409395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.409382    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386c604-04c2-4d21-bd0d-99b4bc965c3e-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.409494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.409480    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5kp\" (UniqueName: \"kubernetes.io/projected/d9a126cd-c401-485f-90f9-eaeb57262e03-kube-api-access-zf5kp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.409618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.409604    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqdr\" (UniqueName: \"kubernetes.io/projected/2d0566a2-4971-4901-a926-af80b11a0603-kube-api-access-gsqdr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.463636 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:22.463560    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:15:22.635691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.635490    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t" event={"ID":"fbc01256-4c3b-40ce-9b18-7b20596c7255","Type":"ContainerDied","Data":"cd5e9d3c5d1bab2635e27e500b118b008b19f89fb8ed9a88d158ab418cc0de50"}
Nov 03 11:15:22.635691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.635537    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"
Nov 03 11:15:22.635691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.635672    5817 scope.go:122] "RemoveContainer" containerID="e0c35f76d6f22cc3879baddaf4659ede09239d13417b92190c5fb4ae6f27d384"
Nov 03 11:15:22.641042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.641022    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"
Nov 03 11:15:22.641042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.641033    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc" event={"ID":"f0967a8b-43f1-496a-8a5c-03e435f50232","Type":"ContainerDied","Data":"19cf371697ea24677420a7e6013f717f93abcc64fd65c7764909ffc1df672721"}
Nov 03 11:15:22.646751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.646726    5817 generic.go:356] "Generic (PLEG): container finished" podID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerID="b4f5b267d1548e5a37412da3a288e78c12789873b91bbd09fd39852692fa2dfa" exitCode=60
Nov 03 11:15:22.646864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.646755    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" event={"ID":"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9","Type":"ContainerDied","Data":"b4f5b267d1548e5a37412da3a288e78c12789873b91bbd09fd39852692fa2dfa"}
Nov 03 11:15:22.647275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.647255    5817 scope.go:122] "RemoveContainer" containerID="b4f5b267d1548e5a37412da3a288e78c12789873b91bbd09fd39852692fa2dfa"
Nov 03 11:15:22.647797 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:22.647604    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895075-tvldd_odh-ai-gateway-infra(bcbcd43e-9753-49ef-b5ad-ab1770a65cc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9"
Nov 03 11:15:22.650772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.650753    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"
Nov 03 11:15:22.650869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.650797    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl" event={"ID":"d9a126cd-c401-485f-90f9-eaeb57262e03","Type":"ContainerDied","Data":"5ff6d6025f950840b0a6d20617e35e0307cd7619da1bfb842e8e4a668e07ec29"}
Nov 03 11:15:22.652588 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.652552    5817 scope.go:122] "RemoveContainer" containerID="2f728ba6adf1a34487556156316b08a0d618a72723e51372343c95cce801a30c"
Nov 03 11:15:22.653329 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.652988    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"
Nov 03 11:15:22.653329 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.652989    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs" event={"ID":"a386c604-04c2-4d21-bd0d-99b4bc965c3e","Type":"ContainerDied","Data":"2388f0d49c2c35df77189823e756e556f0c1aaf60a11ff6f7a0492968e8f5311"}
Nov 03 11:15:22.655021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.655002    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd" event={"ID":"2d0566a2-4971-4901-a926-af80b11a0603","Type":"ContainerDied","Data":"af4a15d17463007f4c92dd8407f3c9245117029a5254196fb8523d9bef03ad2d"}
Nov 03 11:15:22.655138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.655028    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"
Nov 03 11:15:22.666794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.666774    5817 scope.go:122] "RemoveContainer" containerID="0e5e1dc6aece02b659dc54fcb403ff720e01d6bcc5d5736f53bf7cdb3f7d9279"
Nov 03 11:15:22.683744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.683721    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"]
Nov 03 11:15:22.685356 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.685335    5817 scope.go:122] "RemoveContainer" containerID="42eb52716df1369ab3ca8bf6ee42cf8b00d2564ea8deabbe1f70123a9f5dbe5e"
Nov 03 11:15:22.694710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.694689    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895075-zs52t"]
Nov 03 11:15:22.701785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.701755    5817 scope.go:122] "RemoveContainer" containerID="2dd202591a4992b34a7105007da40a2f55f0f5ff10e189fbd0ca3db9960af5ac"
Nov 03 11:15:22.706702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.706681    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"]
Nov 03 11:15:22.714878 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.714855    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895075-4kwfl"]
Nov 03 11:15:22.718092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.718073    5817 scope.go:122] "RemoveContainer" containerID="0665d01dcf9e189b1751be5ec79f68db41cb04b1988563de4d7e4fd813b37f95"
Nov 03 11:15:22.735188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.734840    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"]
Nov 03 11:15:22.740225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.740195    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9265bf-29895075-hklbc"]
Nov 03 11:15:22.748448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.748426    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"]
Nov 03 11:15:22.756396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.756374    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-60ad03-29895075-s6wzd"]
Nov 03 11:15:22.760609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.760588    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"]
Nov 03 11:15:22.764908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:22.764886    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895075-x5zrs"]
Nov 03 11:15:23.187309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.187276    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:23.676960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.676877    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"]
Nov 03 11:15:23.881597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.877691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:23.967347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.964793    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" path="/var/lib/kubelet/pods/154bc19a-4ea4-47ea-9a6a-d5c3920f902d/volumes"
Nov 03 11:15:23.972634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.970472    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0566a2-4971-4901-a926-af80b11a0603" path="/var/lib/kubelet/pods/2d0566a2-4971-4901-a926-af80b11a0603/volumes"
Nov 03 11:15:23.972634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.971139    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" path="/var/lib/kubelet/pods/8f21b3aa-337d-46be-ae7a-c5658f48b0af/volumes"
Nov 03 11:15:23.972634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.971649    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" path="/var/lib/kubelet/pods/a386c604-04c2-4d21-bd0d-99b4bc965c3e/volumes"
Nov 03 11:15:23.973337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.973303    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" path="/var/lib/kubelet/pods/d9a126cd-c401-485f-90f9-eaeb57262e03/volumes"
Nov 03 11:15:23.973853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.973837    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" path="/var/lib/kubelet/pods/f0967a8b-43f1-496a-8a5c-03e435f50232/volumes"
Nov 03 11:15:23.974287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.974270    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" path="/var/lib/kubelet/pods/fbc01256-4c3b-40ce-9b18-7b20596c7255/volumes"
Nov 03 11:15:23.998648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:23.998585    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:24.069090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.069050    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-maas-api-service-ca\") pod \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\" (UID: \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\") "
Nov 03 11:15:24.069320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.069285    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-kube-api-access-dq8t2\" (UniqueName: \"kubernetes.io/projected/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-kube-api-access-dq8t2\") pod \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\" (UID: \"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9\") "
Nov 03 11:15:24.069394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.069379    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-maas-api-service-ca" pod "bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" (UID: "bcbcd43e-9753-49ef-b5ad-ab1770a65cc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:24.070348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.070322    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:24.072017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.071989    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-kube-api-access-dq8t2" pod "bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" (UID: "bcbcd43e-9753-49ef-b5ad-ab1770a65cc9"). InnerVolumeSpecName "kube-api-access-dq8t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:24.172024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.171993    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq8t2\" (UniqueName: \"kubernetes.io/projected/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9-kube-api-access-dq8t2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:24.695286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.695245    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd" event={"ID":"bcbcd43e-9753-49ef-b5ad-ab1770a65cc9","Type":"ContainerDied","Data":"25c43d013461bb0f59fcbe65049a05f7df3496d1a85df63e6e0fa71934d1476f"}
Nov 03 11:15:24.695829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.695368    5817 scope.go:122] "RemoveContainer" containerID="b4f5b267d1548e5a37412da3a288e78c12789873b91bbd09fd39852692fa2dfa"
Nov 03 11:15:24.695829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.695544    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"
Nov 03 11:15:24.728109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.728078    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"]
Nov 03 11:15:24.734158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:24.734135    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895075-tvldd"]
Nov 03 11:15:25.576049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:25.576014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:25.708775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:25.708742    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:25.909533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:25.909442    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" path="/var/lib/kubelet/pods/bcbcd43e-9753-49ef-b5ad-ab1770a65cc9/volumes"
Nov 03 11:15:26.003599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:25.989678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:26.669826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:26.669771    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:27.052594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:27.048667    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"]
Nov 03 11:15:27.096609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:27.090929    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70808cef-swvxt"]
Nov 03 11:15:27.965651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:27.962593    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2ab2a2-8efa-48bf-b853-0984cc47acdc" path="/var/lib/kubelet/pods/ff2ab2a2-8efa-48bf-b853-0984cc47acdc/volumes"
Nov 03 11:15:28.010700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:28.010656    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:28.557672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:28.552631    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:29.170600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:29.162770    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:29.841276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:29.841239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:30.475142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:30.475109    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:30.972090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:30.972055    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"]
Nov 03 11:15:30.972486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:30.972300    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy" containerID="cri-o://629c94b0f55c5ff4e3329e753d37344725d16059f939fbd63db25f58a0afbf83" gracePeriod=30
Nov 03 11:15:31.274760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.274633    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:31.367180 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.367140    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"]
Nov 03 11:15:31.407916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.407880    5817 cpu_manager.go:414] "containerMap: removing container" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:31.408171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.408149    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:31.408298 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.408284    5817 cpu_manager.go:414] "containerMap: removing container" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:31.408396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.408384    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:31.408500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.408488    5817 cpu_manager.go:414] "containerMap: removing container" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:31.408618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.408604    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:31.408739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.408727    5817 cpu_manager.go:414] "containerMap: removing container" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.419218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.419195    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.419374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.419361    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.419484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.419472    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.419677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.419660    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:31.419859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.419843    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:31.419985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.419964    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:31.420078 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420067    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:31.420165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420155    5817 cpu_manager.go:414] "containerMap: removing container" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:31.420241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420231    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:31.420336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420326    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:31.420416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420405    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:31.420512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420502    5817 cpu_manager.go:414] "containerMap: removing container" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.420618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420606    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.420717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420707    5817 cpu_manager.go:414] "containerMap: removing container" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:31.420799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420788    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:31.420892 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420881    5817 cpu_manager.go:414] "containerMap: removing container" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:31.420972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.420962    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:31.421113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421100    5817 cpu_manager.go:414] "containerMap: removing container" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.421210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421199    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.421309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421298    5817 cpu_manager.go:414] "containerMap: removing container" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:31.421391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421380    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:31.421483 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421472    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:31.421564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421553    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:31.421701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421689    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.421856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421841    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.421999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.421986    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:31.422996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.422978    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:31.423130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423119    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:31.423224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423212    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:31.423316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423306    5817 cpu_manager.go:414] "containerMap: removing container" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.423407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423396    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.423500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423490    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:31.423602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423590    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:31.423700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423690    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:31.423776 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423766    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:31.423862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423852    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.423946 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.423935    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.424040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424029    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.424120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424110    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.424217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424208    5817 cpu_manager.go:414] "containerMap: removing container" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.424299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424289    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.424390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424380    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.424533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424521    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.424667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424655    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:31.424756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424745    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:31.424849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424838    5817 cpu_manager.go:414] "containerMap: removing container" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:31.424932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.424922    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:31.425031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425021    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:31.425119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425109    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:31.425208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425197    5817 cpu_manager.go:414] "containerMap: removing container" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:31.425290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425280    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:31.425374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425365    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.425452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425443    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.425535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425526    5817 cpu_manager.go:414] "containerMap: removing container" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.425639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425626    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.425780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425766    5817 cpu_manager.go:414] "containerMap: removing container" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.425888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425876    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.425980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.425970    5817 cpu_manager.go:414] "containerMap: removing container" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.426062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426051    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.426160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426150    5817 cpu_manager.go:414] "containerMap: removing container" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.426242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426232    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.426331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426321    5817 cpu_manager.go:414] "containerMap: removing container" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:31.426411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426401    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426488    5817 cpu_manager.go:414] "containerMap: removing container" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426503    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426518    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426525    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426818    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426832    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426842    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426854    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426869    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426880    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426890    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426898    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426907    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426917    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426925    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426936    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426943    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426952    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426962    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426970    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426981    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.426992    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427000    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427009    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427018    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427031    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427041    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427048    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f21b3aa-337d-46be-ae7a-c5658f48b0af" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427057    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427066    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427077    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427089    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="154bc19a-4ea4-47ea-9a6a-d5c3920f902d" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427099    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="24da8686-666d-4c92-81fa-5e6a07e4c5bb" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427108    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427119    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427133    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427141    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427149    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427158    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a386c604-04c2-4d21-bd0d-99b4bc965c3e" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427170    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427179    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:31.429649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.427190    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ae57e1-e067-4165-bd2d-1e4a2ebdfbc6" containerName="cleanup"
Nov 03 11:15:31.451619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.448908    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"]
Nov 03 11:15:31.451619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.448940    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.454095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.454072    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:15:31.454560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.454542    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:15:31.556941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.556850    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.557118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.556979    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.557118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.557010    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2rr\" (UniqueName: \"kubernetes.io/projected/f197cb36-4238-451d-bb21-d06d76a18460-kube-api-access-tn2rr\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.659418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.659378    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.659665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.659455    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.659665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.659479    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2rr\" (UniqueName: \"kubernetes.io/projected/f197cb36-4238-451d-bb21-d06d76a18460-kube-api-access-tn2rr\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.660434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.660398    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.662612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.662494    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.667617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.667586    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2rr\" (UniqueName: \"kubernetes.io/projected/f197cb36-4238-451d-bb21-d06d76a18460-kube-api-access-tn2rr\") pod \"maas-api-revoke-sub-0476e0bd-lcsmf\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:31.767614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:31.767547    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:32.077311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:32.077088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:32.321139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:32.321112    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"]
Nov 03 11:15:32.712603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:32.710634    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:33.121338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:33.121247    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf" event={"ID":"f197cb36-4238-451d-bb21-d06d76a18460","Type":"ContainerStarted","Data":"1e75fb415d1a00379337a0320319c7a7fd04065e301f431c1a2e5a6aa05261a1"}
Nov 03 11:15:33.121338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:33.121314    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf" event={"ID":"f197cb36-4238-451d-bb21-d06d76a18460","Type":"ContainerStarted","Data":"45acbabbbb6f29814b95766fc2d8f1fc8d1cb384975165453815b3591f47a1d7"}
Nov 03 11:15:33.137800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:33.137749    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf" podStartSLOduration=2.137734601 podStartE2EDuration="2.137734601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:33.136201477 +0000 UTC m=+2763.341783525" watchObservedRunningTime="2026-11-03 11:15:33.137734601 +0000 UTC m=+2763.343316651"
Nov 03 11:15:33.406009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:33.405418    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:33.979794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:33.979710    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:15:33.979794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:33.979772    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:15:34.087596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.087092    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:34.148510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.148475    5817 generic.go:356] "Generic (PLEG): container finished" podID="f197cb36-4238-451d-bb21-d06d76a18460" containerID="1e75fb415d1a00379337a0320319c7a7fd04065e301f431c1a2e5a6aa05261a1" exitCode=0
Nov 03 11:15:34.148972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.148631    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf" event={"ID":"f197cb36-4238-451d-bb21-d06d76a18460","Type":"ContainerDied","Data":"1e75fb415d1a00379337a0320319c7a7fd04065e301f431c1a2e5a6aa05261a1"}
Nov 03 11:15:34.372464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.372376    5817 scope.go:122] "RemoveContainer" containerID="3d9b6b60a98df7884a88c8736c2551e5fd3585049bd866400c89bdb8c2a05dc3"
Nov 03 11:15:34.388449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.388427    5817 scope.go:122] "RemoveContainer" containerID="3669ceda47e5b1ca067b5bb29079c2fa5162359213484d6c6c6f435cb5199c4f"
Nov 03 11:15:34.401625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.401602    5817 scope.go:122] "RemoveContainer" containerID="193df6b0762a2e01a5b3fdf570573197b7009ea976aa0bec32e7431797ad702c"
Nov 03 11:15:34.414397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.414376    5817 scope.go:122] "RemoveContainer" containerID="50f510f5fcd187e3a91adf2fb91f4854e2e98c241eaaa7a0783caec4f7578c69"
Nov 03 11:15:34.427689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.427665    5817 scope.go:122] "RemoveContainer" containerID="ab92637dfaf6061fcf63a8289b658f28f43101001969dd4c90925ea3ac0482a6"
Nov 03 11:15:34.774596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.769249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:34.981740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.981702    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:15:34.981908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:34.981750    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:15:35.354471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.354341    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:35.478233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.478201    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:35.515361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.514644    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-openshift-ca\") pod \"f197cb36-4238-451d-bb21-d06d76a18460\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") "
Nov 03 11:15:35.515361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.514815    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-xks-ca\") pod \"f197cb36-4238-451d-bb21-d06d76a18460\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") "
Nov 03 11:15:35.515361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.514986    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f197cb36-4238-451d-bb21-d06d76a18460-kube-api-access-tn2rr\" (UniqueName: \"kubernetes.io/projected/f197cb36-4238-451d-bb21-d06d76a18460-kube-api-access-tn2rr\") pod \"f197cb36-4238-451d-bb21-d06d76a18460\" (UID: \"f197cb36-4238-451d-bb21-d06d76a18460\") "
Nov 03 11:15:35.517997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.517963    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-openshift-ca" pod "f197cb36-4238-451d-bb21-d06d76a18460" (UID: "f197cb36-4238-451d-bb21-d06d76a18460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:35.527708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.522033    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-xks-ca" pod "f197cb36-4238-451d-bb21-d06d76a18460" (UID: "f197cb36-4238-451d-bb21-d06d76a18460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:15:35.540804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.540770    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f197cb36-4238-451d-bb21-d06d76a18460-kube-api-access-tn2rr" pod "f197cb36-4238-451d-bb21-d06d76a18460" (UID: "f197cb36-4238-451d-bb21-d06d76a18460"). InnerVolumeSpecName "kube-api-access-tn2rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:35.618397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.618354    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:35.618397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.618390    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f197cb36-4238-451d-bb21-d06d76a18460-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:35.618397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:35.618405    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2rr\" (UniqueName: \"kubernetes.io/projected/f197cb36-4238-451d-bb21-d06d76a18460-kube-api-access-tn2rr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.086612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.078914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:36.186619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.186316    5817 generic.go:356] "Generic (PLEG): container finished" podID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerID="629c94b0f55c5ff4e3329e753d37344725d16059f939fbd63db25f58a0afbf83" exitCode=0
Nov 03 11:15:36.187065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.186553    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" event={"ID":"76c730b7-3e59-4d9b-a03f-ff1fab06aaed","Type":"ContainerDied","Data":"629c94b0f55c5ff4e3329e753d37344725d16059f939fbd63db25f58a0afbf83"}
Nov 03 11:15:36.187065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.186843    5817 scope.go:122] "RemoveContainer" containerID="bcc018edb94040e6656fe04c887ee40e4e495f3716cfe0104fa67b13dbab3008"
Nov 03 11:15:36.194423 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.194394    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf" event={"ID":"f197cb36-4238-451d-bb21-d06d76a18460","Type":"ContainerDied","Data":"45acbabbbb6f29814b95766fc2d8f1fc8d1cb384975165453815b3591f47a1d7"}
Nov 03 11:15:36.194522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.194454    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45acbabbbb6f29814b95766fc2d8f1fc8d1cb384975165453815b3591f47a1d7"
Nov 03 11:15:36.194615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.194525    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"
Nov 03 11:15:36.348504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.348469    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 11:15:36.445711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.445672    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-socket\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.446196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.445739    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istiod-ca-cert\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.446196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.445786    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-kube-api-access-8422c\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-kube-api-access-8422c\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.446196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.445901    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-envoy\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.446196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.446008    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-socket" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:15:36.446196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.446062    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-token\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-token\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.446196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.446089    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-certs\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.446557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.446291    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-data\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.446557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.446419    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-credential-socket\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.446718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.446656    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-podinfo\") pod \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\" (UID: \"76c730b7-3e59-4d9b-a03f-ff1fab06aaed\") "
Nov 03 11:15:36.447589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.446908    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-data" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:15:36.447718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.447115    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-credential-socket" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:15:36.447718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.447182    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-certs" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:15:36.447718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.447386    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istiod-ca-cert" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:36.448209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.448182    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.448303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.448216    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.448303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.448232    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.448303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.448244    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.448303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.448257    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.451865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.450806    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-podinfo" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:15:36.451865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.450845    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-kube-api-access-8422c" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "kube-api-access-8422c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:36.452602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.452553    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-token" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:36.453505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.453477    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-envoy" pod "76c730b7-3e59-4d9b-a03f-ff1fab06aaed" (UID: "76c730b7-3e59-4d9b-a03f-ff1fab06aaed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:15:36.550880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.550844    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.551049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.550884    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8422c\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-kube-api-access-8422c\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.551049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.550902    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.551049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.550916    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c730b7-3e59-4d9b-a03f-ff1fab06aaed-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:36.861599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:36.851880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:37.210564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:37.210530    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"
Nov 03 11:15:37.210564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:37.210548    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r" event={"ID":"76c730b7-3e59-4d9b-a03f-ff1fab06aaed","Type":"ContainerDied","Data":"5cc609b5740ff38605cddccce951d67f211962949ec63268de0e381de6032dbc"}
Nov 03 11:15:37.210904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:37.210662    5817 scope.go:122] "RemoveContainer" containerID="629c94b0f55c5ff4e3329e753d37344725d16059f939fbd63db25f58a0afbf83"
Nov 03 11:15:37.242669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:37.242630    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"]
Nov 03 11:15:37.247780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:37.247755    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-vd55r"]
Nov 03 11:15:37.577644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:37.576283    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:37.912009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:37.911495    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" path="/var/lib/kubelet/pods/76c730b7-3e59-4d9b-a03f-ff1fab06aaed/volumes"
Nov 03 11:15:38.213001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:38.212970    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:39.098595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.091870    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:39.266078 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.266037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:39.308977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.308945    5817 generic.go:356] "Generic (PLEG): container finished" podID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerID="a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867" exitCode=0
Nov 03 11:15:39.309150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.308987    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" event={"ID":"a9fce276-e6fe-4835-b146-c9a4642598e5","Type":"ContainerDied","Data":"a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867"}
Nov 03 11:15:39.309845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.309722    5817 scope.go:122] "RemoveContainer" containerID="a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867"
Nov 03 11:15:39.313069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.312986    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:39.313184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.313076    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:39.313184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.313120    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:39.501014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:39.500976    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:40.320153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:40.320099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:40.345762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:40.345725    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" event={"ID":"a9fce276-e6fe-4835-b146-c9a4642598e5","Type":"ContainerStarted","Data":"d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835"}
Nov 03 11:15:40.626744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:40.626659    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:15:40.627440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:40.627417    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 11:15:40.627523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:40.627465    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 11:15:40.861435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:40.861133    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:41.626874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.626833    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 11:15:41.627364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.626911    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 11:15:41.744117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.737636    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:41.796971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.796926    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"]
Nov 03 11:15:41.798518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.798489    5817 cpu_manager.go:414] "containerMap: removing container" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy"
Nov 03 11:15:41.798710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.798693    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy"
Nov 03 11:15:41.798826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.798812    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:41.798923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.798910    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:41.799041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799028    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:41.799149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799136    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0566a2-4971-4901-a926-af80b11a0603" containerName="cleanup"
Nov 03 11:15:41.799261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799249    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f197cb36-4238-451d-bb21-d06d76a18460" containerName="revoke-keys"
Nov 03 11:15:41.799354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799341    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f197cb36-4238-451d-bb21-d06d76a18460" containerName="revoke-keys"
Nov 03 11:15:41.799454 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799442    5817 cpu_manager.go:414] "containerMap: removing container" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:41.799539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799529    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:41.799648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799636    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:41.799748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799736    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a126cd-c401-485f-90f9-eaeb57262e03" containerName="cleanup"
Nov 03 11:15:41.799857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799845    5817 cpu_manager.go:414] "containerMap: removing container" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:41.799941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.799931    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:41.800064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.800052    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:41.800147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.800136    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0967a8b-43f1-496a-8a5c-03e435f50232" containerName="cleanup"
Nov 03 11:15:41.800245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.800235    5817 cpu_manager.go:414] "containerMap: removing container" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:41.800331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.800321    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:41.800443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.800432    5817 cpu_manager.go:414] "containerMap: removing container" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:41.800527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.800516    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:41.800873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.800855    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="90118aeb-a009-4ce0-b6dd-81c1e08b26e9" containerName="cleanup"
Nov 03 11:15:41.800990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.800977    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f197cb36-4238-451d-bb21-d06d76a18460" containerName="revoke-keys"
Nov 03 11:15:41.801084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.801074    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy"
Nov 03 11:15:41.801181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.801171    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc01256-4c3b-40ce-9b18-7b20596c7255" containerName="cleanup"
Nov 03 11:15:41.801272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.801263    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbcd43e-9753-49ef-b5ad-ab1770a65cc9" containerName="cleanup"
Nov 03 11:15:41.801352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.801342    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec82e8-8882-4d0a-bcfe-9ad2c0b3f464" containerName="cleanup"
Nov 03 11:15:41.801441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.801430    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="38af2eca-791e-4da1-a74d-f93b4a6fa8e9" containerName="cleanup"
Nov 03 11:15:41.801530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.801520    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a0333b-9ed5-4e5a-95cf-ea734272085b" containerName="cleanup"
Nov 03 11:15:41.801648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.801637    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da14868-435e-4ed1-96a3-b295fa1ac1a1" containerName="cleanup"
Nov 03 11:15:41.854590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.843312    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"]
Nov 03 11:15:41.854590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.844171    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:41.863472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.854884    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:15:41.863472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.856002    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:15:41.915639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.914751    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnp7z\" (UniqueName: \"kubernetes.io/projected/81cac2d4-6200-4637-bbd8-28d850759492-kube-api-access-fnp7z\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:41.915639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.914835    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:41.915639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:41.915184    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:42.019765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.019712    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:42.019979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.019848    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnp7z\" (UniqueName: \"kubernetes.io/projected/81cac2d4-6200-4637-bbd8-28d850759492-kube-api-access-fnp7z\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:42.019979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.019910    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:42.020589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.020550    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:42.023626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.023600    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:42.028194 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.028171    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnp7z\" (UniqueName: \"kubernetes.io/projected/81cac2d4-6200-4637-bbd8-28d850759492-kube-api-access-fnp7z\") pod \"maas-api-revoke-sub-5e4a224c-l8rqg\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:42.179190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.178522    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:42.480780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.480639    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:42.580190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.580150    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"]
Nov 03 11:15:42.580931 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:42.580904    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cac2d4_6200_4637_bbd8_28d850759492.slice/crio-859c4a40ef753ee837eabc41623bdb765f202800cfb5d0d7524e6a146a5271eb WatchSource:0}: Error finding container 859c4a40ef753ee837eabc41623bdb765f202800cfb5d0d7524e6a146a5271eb: Status 404 returned error can't find the container with id 859c4a40ef753ee837eabc41623bdb765f202800cfb5d0d7524e6a146a5271eb
Nov 03 11:15:42.628632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.627800    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 11:15:42.628632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.627845    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 11:15:42.681660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:42.674322    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:43.120334 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.120266    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:43.266246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.266193    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:43.376626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.376587    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg" event={"ID":"81cac2d4-6200-4637-bbd8-28d850759492","Type":"ContainerStarted","Data":"1304877d6df360e85279a1d172cb92cd4fb16881f2bf2a279f2546feea311863"}
Nov 03 11:15:43.376804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.376649    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg" event={"ID":"81cac2d4-6200-4637-bbd8-28d850759492","Type":"ContainerStarted","Data":"859c4a40ef753ee837eabc41623bdb765f202800cfb5d0d7524e6a146a5271eb"}
Nov 03 11:15:43.419596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.416849    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg" podStartSLOduration=2.416828664 podStartE2EDuration="2.416828664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:43.40248912 +0000 UTC m=+2773.608071172" watchObservedRunningTime="2026-11-03 11:15:43.416828664 +0000 UTC m=+2773.622410718"
Nov 03 11:15:43.617650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.614704    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:43.664618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.649040    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 11:15:43.664618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.649116    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 11:15:43.776669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:43.776637    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:44.514596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:44.510739    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:44.525402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:44.524756    5817 generic.go:356] "Generic (PLEG): container finished" podID="81cac2d4-6200-4637-bbd8-28d850759492" containerID="1304877d6df360e85279a1d172cb92cd4fb16881f2bf2a279f2546feea311863" exitCode=0
Nov 03 11:15:44.525402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:44.524960    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg" event={"ID":"81cac2d4-6200-4637-bbd8-28d850759492","Type":"ContainerDied","Data":"1304877d6df360e85279a1d172cb92cd4fb16881f2bf2a279f2546feea311863"}
Nov 03 11:15:44.627027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:44.626941    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 11:15:44.627027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:44.627006    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 11:15:45.248450 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.248415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:45.627230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.627125    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 11:15:45.627230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.627171    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 11:15:45.807647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.799756    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:45.873322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.873292    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cac2d4-6200-4637-bbd8-28d850759492-kube-api-access-fnp7z\" (UniqueName: \"kubernetes.io/projected/81cac2d4-6200-4637-bbd8-28d850759492-kube-api-access-fnp7z\") pod \"81cac2d4-6200-4637-bbd8-28d850759492\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") "
Nov 03 11:15:45.873488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.873449    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-xks-ca\") pod \"81cac2d4-6200-4637-bbd8-28d850759492\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") "
Nov 03 11:15:45.873752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.873726    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-openshift-ca\") pod \"81cac2d4-6200-4637-bbd8-28d850759492\" (UID: \"81cac2d4-6200-4637-bbd8-28d850759492\") "
Nov 03 11:15:45.874258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.874235    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-openshift-ca" pod "81cac2d4-6200-4637-bbd8-28d850759492" (UID: "81cac2d4-6200-4637-bbd8-28d850759492"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:45.874537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.874498    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:45.875934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.875906    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-xks-ca" pod "81cac2d4-6200-4637-bbd8-28d850759492" (UID: "81cac2d4-6200-4637-bbd8-28d850759492"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:15:45.876508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.876486    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cac2d4-6200-4637-bbd8-28d850759492-kube-api-access-fnp7z" pod "81cac2d4-6200-4637-bbd8-28d850759492" (UID: "81cac2d4-6200-4637-bbd8-28d850759492"). InnerVolumeSpecName "kube-api-access-fnp7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:45.964604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.964490    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:45.981134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.977463    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnp7z\" (UniqueName: \"kubernetes.io/projected/81cac2d4-6200-4637-bbd8-28d850759492-kube-api-access-fnp7z\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:45.981134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:45.977498    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cac2d4-6200-4637-bbd8-28d850759492-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:46.547320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:46.547296    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"
Nov 03 11:15:46.547983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:46.547284    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg" event={"ID":"81cac2d4-6200-4637-bbd8-28d850759492","Type":"ContainerDied","Data":"859c4a40ef753ee837eabc41623bdb765f202800cfb5d0d7524e6a146a5271eb"}
Nov 03 11:15:46.547983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:46.547445    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="859c4a40ef753ee837eabc41623bdb765f202800cfb5d0d7524e6a146a5271eb"
Nov 03 11:15:46.627174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:46.627138    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 11:15:46.627360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:46.627196    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 11:15:46.751598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:46.746732    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:47.634062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:47.634031    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:15:47.634546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:47.634445    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:15:47.635249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:47.635220    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:15:47.649621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:47.649564    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:47.788552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:47.788519    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:48.724095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.724067    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:48.890865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.890829    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:48.899712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.899683    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"]
Nov 03 11:15:48.900541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.900526    5817 cpu_manager.go:414] "containerMap: removing container" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy"
Nov 03 11:15:48.900613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.900543    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy"
Nov 03 11:15:48.900613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.900559    5817 cpu_manager.go:414] "containerMap: removing container" podUID="81cac2d4-6200-4637-bbd8-28d850759492" containerName="revoke-keys"
Nov 03 11:15:48.900613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.900566    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cac2d4-6200-4637-bbd8-28d850759492" containerName="revoke-keys"
Nov 03 11:15:48.900754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.900744    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cac2d4-6200-4637-bbd8-28d850759492" containerName="revoke-keys"
Nov 03 11:15:48.900792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.900760    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c730b7-3e59-4d9b-a03f-ff1fab06aaed" containerName="istio-proxy"
Nov 03 11:15:48.905030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.905007    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:48.910025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.909989    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-24376f\""
Nov 03 11:15:48.910200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.910183    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-24376f-dockercfg-dsm8q\""
Nov 03 11:15:48.918928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.918906    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"]
Nov 03 11:15:48.940450 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.940420    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"]
Nov 03 11:15:48.947966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.947941    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"]
Nov 03 11:15:48.948450 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:48.948430    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:49.112645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.110907    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6pm\" (UniqueName: \"kubernetes.io/projected/544fda18-938c-4239-82dc-9920f64e3938-kube-api-access-pr6pm\") pod \"payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d\" (UID: \"544fda18-938c-4239-82dc-9920f64e3938\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:49.112645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.111063    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwst\" (UniqueName: \"kubernetes.io/projected/b7f3810a-c5ff-41bf-bd62-8767c026017b-kube-api-access-4jwst\") pod \"payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk\" (UID: \"b7f3810a-c5ff-41bf-bd62-8767c026017b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:49.112645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.111111    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fda18-938c-4239-82dc-9920f64e3938-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d\" (UID: \"544fda18-938c-4239-82dc-9920f64e3938\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:49.112645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.111285    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f3810a-c5ff-41bf-bd62-8767c026017b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk\" (UID: \"b7f3810a-c5ff-41bf-bd62-8767c026017b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:49.164605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.161135    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:49.229601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.227817    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f3810a-c5ff-41bf-bd62-8767c026017b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk\" (UID: \"b7f3810a-c5ff-41bf-bd62-8767c026017b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:49.229601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.228114    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6pm\" (UniqueName: \"kubernetes.io/projected/544fda18-938c-4239-82dc-9920f64e3938-kube-api-access-pr6pm\") pod \"payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d\" (UID: \"544fda18-938c-4239-82dc-9920f64e3938\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:49.229601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.228232    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwst\" (UniqueName: \"kubernetes.io/projected/b7f3810a-c5ff-41bf-bd62-8767c026017b-kube-api-access-4jwst\") pod \"payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk\" (UID: \"b7f3810a-c5ff-41bf-bd62-8767c026017b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:49.229601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.228273    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fda18-938c-4239-82dc-9920f64e3938-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d\" (UID: \"544fda18-938c-4239-82dc-9920f64e3938\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:49.229601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.229227    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fda18-938c-4239-82dc-9920f64e3938-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d\" (UID: \"544fda18-938c-4239-82dc-9920f64e3938\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:49.259621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.252161    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f3810a-c5ff-41bf-bd62-8767c026017b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk\" (UID: \"b7f3810a-c5ff-41bf-bd62-8767c026017b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:49.368189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.368050    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwst\" (UniqueName: \"kubernetes.io/projected/b7f3810a-c5ff-41bf-bd62-8767c026017b-kube-api-access-4jwst\") pod \"payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk\" (UID: \"b7f3810a-c5ff-41bf-bd62-8767c026017b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:49.384665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.369789    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6pm\" (UniqueName: \"kubernetes.io/projected/544fda18-938c-4239-82dc-9920f64e3938-kube-api-access-pr6pm\") pod \"payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d\" (UID: \"544fda18-938c-4239-82dc-9920f64e3938\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:49.421336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.421075    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:49.527048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.527012    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:49.657676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.656691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:49.759689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.759611    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"]
Nov 03 11:15:49.760162 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:49.760141    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544fda18_938c_4239_82dc_9920f64e3938.slice/crio-db5d6e52a55d7dc556afedb72eeb9e3841cd256b37d9655da492e1ed72bef17e WatchSource:0}: Error finding container db5d6e52a55d7dc556afedb72eeb9e3841cd256b37d9655da492e1ed72bef17e: Status 404 returned error can't find the container with id db5d6e52a55d7dc556afedb72eeb9e3841cd256b37d9655da492e1ed72bef17e
Nov 03 11:15:49.826607 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:15:49.826546    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f3810a_c5ff_41bf_bd62_8767c026017b.slice/crio-2fe4cc0e5eceab9ce81425bda301f5eafa00563461458d707c1dc068f6d23a9c WatchSource:0}: Error finding container 2fe4cc0e5eceab9ce81425bda301f5eafa00563461458d707c1dc068f6d23a9c: Status 404 returned error can't find the container with id 2fe4cc0e5eceab9ce81425bda301f5eafa00563461458d707c1dc068f6d23a9c
Nov 03 11:15:49.828517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:49.828492    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"]
Nov 03 11:15:50.303319 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.303288    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:50.596409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.596306    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d" event={"ID":"544fda18-938c-4239-82dc-9920f64e3938","Type":"ContainerStarted","Data":"d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda"}
Nov 03 11:15:50.596409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.596378    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d" event={"ID":"544fda18-938c-4239-82dc-9920f64e3938","Type":"ContainerStarted","Data":"db5d6e52a55d7dc556afedb72eeb9e3841cd256b37d9655da492e1ed72bef17e"}
Nov 03 11:15:50.596409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.596408    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:15:50.598439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.598413    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk" event={"ID":"b7f3810a-c5ff-41bf-bd62-8767c026017b","Type":"ContainerStarted","Data":"a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24"}
Nov 03 11:15:50.598609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.598458    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk" event={"ID":"b7f3810a-c5ff-41bf-bd62-8767c026017b","Type":"ContainerStarted","Data":"2fe4cc0e5eceab9ce81425bda301f5eafa00563461458d707c1dc068f6d23a9c"}
Nov 03 11:15:50.598609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.598479    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:15:50.619061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.618386    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d" podStartSLOduration=2.618373272 podStartE2EDuration="2.618373272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:50.611460634 +0000 UTC m=+2780.817042686" watchObservedRunningTime="2026-11-03 11:15:50.618373272 +0000 UTC m=+2780.823955371"
Nov 03 11:15:50.632729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:50.632687    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk" podStartSLOduration=2.632674069 podStartE2EDuration="2.632674069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:50.625924421 +0000 UTC m=+2780.831506476" watchObservedRunningTime="2026-11-03 11:15:50.632674069 +0000 UTC m=+2780.838256266"
Nov 03 11:15:51.109255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:51.107531    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:51.662601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:51.660333    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:52.341599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:52.331756    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:52.463109 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:52.463074    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:15:53.075701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:53.074651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:53.211597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:53.209358    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:15:53.305453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:53.305414    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:53.483600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:53.473756    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:53.688190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:53.686827    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"]
Nov 03 11:15:53.824909 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:53.824808    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:54.151695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:54.138953    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:54.333596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:54.325392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:54.539617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:54.539555    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:55.587802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:55.586112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:56.316600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:56.314525    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:57.108748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:57.098247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:57.905602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:57.894968    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:58.692724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:58.692647    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp" podUID="37b083a5-e5e2-45e9-a598-85db51df4429" containerName="payload-processing" containerID="cri-o://700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822" gracePeriod=25
Nov 03 11:15:58.756444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:58.756407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:59.057556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.057518    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:15:59.137779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.137740    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-kube-api-access-kdckc\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-kube-api-access-kdckc\") pod \"37b083a5-e5e2-45e9-a598-85db51df4429\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") "
Nov 03 11:15:59.137987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.137922    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-plugins-config-volume\") pod \"37b083a5-e5e2-45e9-a598-85db51df4429\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") "
Nov 03 11:15:59.137987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.137971    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-routing-overlay-volume\") pod \"37b083a5-e5e2-45e9-a598-85db51df4429\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") "
Nov 03 11:15:59.138075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.138049    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-provider-credentials\") pod \"37b083a5-e5e2-45e9-a598-85db51df4429\" (UID: \"37b083a5-e5e2-45e9-a598-85db51df4429\") "
Nov 03 11:15:59.138334 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.138304    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-plugins-config-volume" pod "37b083a5-e5e2-45e9-a598-85db51df4429" (UID: "37b083a5-e5e2-45e9-a598-85db51df4429"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:59.138449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.138430    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-routing-overlay-volume" pod "37b083a5-e5e2-45e9-a598-85db51df4429" (UID: "37b083a5-e5e2-45e9-a598-85db51df4429"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:59.139074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.139050    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:59.139074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.139073    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b083a5-e5e2-45e9-a598-85db51df4429-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:59.140602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.140552    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-kube-api-access-kdckc" pod "37b083a5-e5e2-45e9-a598-85db51df4429" (UID: "37b083a5-e5e2-45e9-a598-85db51df4429"). InnerVolumeSpecName "kube-api-access-kdckc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:59.140706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.140652    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-provider-credentials" pod "37b083a5-e5e2-45e9-a598-85db51df4429" (UID: "37b083a5-e5e2-45e9-a598-85db51df4429"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:59.240891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.240856    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:59.240891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.240889    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdckc\" (UniqueName: \"kubernetes.io/projected/37b083a5-e5e2-45e9-a598-85db51df4429-kube-api-access-kdckc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:59.479623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.478470    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:15:59.811658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.810936    5817 generic.go:356] "Generic (PLEG): container finished" podID="37b083a5-e5e2-45e9-a598-85db51df4429" containerID="700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822" exitCode=0
Nov 03 11:15:59.811658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.811299    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp" event={"ID":"37b083a5-e5e2-45e9-a598-85db51df4429","Type":"ContainerDied","Data":"700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822"}
Nov 03 11:15:59.811658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.811354    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp" event={"ID":"37b083a5-e5e2-45e9-a598-85db51df4429","Type":"ContainerDied","Data":"c4690b78211a701a3d7a55ab6d775a2bd8efb5efdcae0a72ccd2eb9b5d81d60c"}
Nov 03 11:15:59.811658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.811390    5817 scope.go:122] "RemoveContainer" containerID="700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822"
Nov 03 11:15:59.811658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.811611    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"
Nov 03 11:15:59.884799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.884769    5817 scope.go:122] "RemoveContainer" containerID="700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822"
Nov 03 11:15:59.885171 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:15:59.885144    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822\": container with ID starting with 700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822 not found: ID does not exist" containerID="700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822"
Nov 03 11:15:59.885286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.885185    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822"} err="failed to get container status \"700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822\": rpc error: code = NotFound desc = could not find container \"700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822\": container with ID starting with 700eaeeab0692c7ee26cb9d97ec8a69916bcec114a5669b8b3edd45435cc2822 not found: ID does not exist"
Nov 03 11:15:59.886476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.886449    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"]
Nov 03 11:15:59.892938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.892918    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-5cbfwxlqp"]
Nov 03 11:15:59.906786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:15:59.906759    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b083a5-e5e2-45e9-a598-85db51df4429" path="/var/lib/kubelet/pods/37b083a5-e5e2-45e9-a598-85db51df4429/volumes"
Nov 03 11:16:00.212594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.201438    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:00.346748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.346716    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:00.482800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.482708    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:00.654754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.654719    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:00.873838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.873751    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"]
Nov 03 11:16:00.874768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.874749    5817 cpu_manager.go:414] "containerMap: removing container" podUID="37b083a5-e5e2-45e9-a598-85db51df4429" containerName="payload-processing"
Nov 03 11:16:00.874824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.874770    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b083a5-e5e2-45e9-a598-85db51df4429" containerName="payload-processing"
Nov 03 11:16:00.874914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.874905    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b083a5-e5e2-45e9-a598-85db51df4429" containerName="payload-processing"
Nov 03 11:16:00.880470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.880153    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:00.885278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.885250    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"]
Nov 03 11:16:00.888624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.888550    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:16:00.888624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.888550    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:16:00.984614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.984545    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:00.984819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.984738    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:00.985258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.985228    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8d6\" (UniqueName: \"kubernetes.io/projected/2b210b14-173e-44af-a9c1-e186d9572370-kube-api-access-js8d6\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:01.015592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:00.998086    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"]
Nov 03 11:16:01.026642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.018618    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"]
Nov 03 11:16:01.026642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.018794    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.027292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.027266    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-68aa46-openshift-default-dockercfg-tf598\""
Nov 03 11:16:01.087025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.086995    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-istio-token\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.087212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087065    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8d6\" (UniqueName: \"kubernetes.io/projected/2b210b14-173e-44af-a9c1-e186d9572370-kube-api-access-js8d6\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:01.087212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087145    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-data\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.087351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087217    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:01.087351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087248    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-envoy\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.087351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087320    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-socket\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.087520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087392    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:01.087644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087617    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f823003-51a6-438f-8270-5fe47685aa9e-istio-podinfo\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.087732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087665    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-credential-socket\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.088018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.087994    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-certs\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.088393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.088366    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b49j8\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-kube-api-access-b49j8\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.088670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.088647    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f823003-51a6-438f-8270-5fe47685aa9e-istiod-ca-cert\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.088846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.088405    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:01.096222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.096194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:01.096415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.096322    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:01.099071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.099044    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8d6\" (UniqueName: \"kubernetes.io/projected/2b210b14-173e-44af-a9c1-e186d9572370-kube-api-access-js8d6\") pod \"maas-api-revoke-sub-d72bc277-mj6xl\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:01.190761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.190731    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49j8\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-kube-api-access-b49j8\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.190916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.190797    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f823003-51a6-438f-8270-5fe47685aa9e-istiod-ca-cert\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.190987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.190924    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-istio-token\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.191050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.190998    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-data\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.191117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191066    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-envoy\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.191184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191137    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-socket\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.191248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191181    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f823003-51a6-438f-8270-5fe47685aa9e-istio-podinfo\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.191248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191214    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-credential-socket\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.191360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191338    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-certs\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.191425 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191386    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-data\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.191600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191553    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f823003-51a6-438f-8270-5fe47685aa9e-istiod-ca-cert\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.192193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191843    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-credential-socket\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.192193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.191918    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-certs\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.192193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.192007    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-socket\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.194034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.193963    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-envoy\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.194462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.194432    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f823003-51a6-438f-8270-5fe47685aa9e-istio-podinfo\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.224360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.212859    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:01.252620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.239420    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-istio-token\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.252620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.251335    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49j8\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-kube-api-access-b49j8\") pod \"e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") " pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.352604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.350865    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:01.355703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.355098    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:01.624609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.620836    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:16:01.690222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.688616    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:16:01.872104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.872059    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"]
Nov 03 11:16:01.877526 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:16:01.877493    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b210b14_173e_44af_a9c1_e186d9572370.slice/crio-ebb2dc7780582bb16a75fbeb47df89df4f6bc4217a048ee8d87a3556d5f82cd1 WatchSource:0}: Error finding container ebb2dc7780582bb16a75fbeb47df89df4f6bc4217a048ee8d87a3556d5f82cd1: Status 404 returned error can't find the container with id ebb2dc7780582bb16a75fbeb47df89df4f6bc4217a048ee8d87a3556d5f82cd1
Nov 03 11:16:01.960351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.960317    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"]
Nov 03 11:16:01.967795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.967764    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:01.967886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.967828    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:01.967886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:01.967856    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:02.170098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.167030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:02.763493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.763307    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:02.814834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.814798    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"]
Nov 03 11:16:02.826956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.826927    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.834068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.833415    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-055d82-openshift-default-dockercfg-fpkbq\""
Nov 03 11:16:02.842841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.842765    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"]
Nov 03 11:16:02.851938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.851800    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.857442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.857304    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"]
Nov 03 11:16:02.871155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.869502    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"]
Nov 03 11:16:02.900675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.900643    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl" event={"ID":"2b210b14-173e-44af-a9c1-e186d9572370","Type":"ContainerStarted","Data":"54ca5f8d01f229c78eba31786752fea9bafa221b44102b63c0f6635f25dc9988"}
Nov 03 11:16:02.901206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.901165    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl" event={"ID":"2b210b14-173e-44af-a9c1-e186d9572370","Type":"ContainerStarted","Data":"ebb2dc7780582bb16a75fbeb47df89df4f6bc4217a048ee8d87a3556d5f82cd1"}
Nov 03 11:16:02.903717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.903691    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" event={"ID":"6f823003-51a6-438f-8270-5fe47685aa9e","Type":"ContainerStarted","Data":"59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92"}
Nov 03 11:16:02.903849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.903739    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" event={"ID":"6f823003-51a6-438f-8270-5fe47685aa9e","Type":"ContainerStarted","Data":"ab37fc758981128742e9dec0d2e434b187008f200d9ef18ed497dc0fa1ebce8b"}
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913420    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913492    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913530    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913590    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913625    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913714    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913757    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngfc\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-kube-api-access-mngfc\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913791    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743f219-4ae7-463e-956b-97e3ec89561f-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913836    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743f219-4ae7-463e-956b-97e3ec89561f-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913867    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bed6f31-3335-410e-a924-a345d610e597-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913896    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.913950    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.914027    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.914065    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bed6f31-3335-410e-a924-a345d610e597-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.914132    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.914189    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4xv\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-kube-api-access-5c4xv\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.914227    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:02.914651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.914280    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:02.920803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.918317    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl" podStartSLOduration=2.918305511 podStartE2EDuration="2.918305511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:02.916163405 +0000 UTC m=+2793.121745456" watchObservedRunningTime="2026-11-03 11:16:02.918305511 +0000 UTC m=+2793.123887562"
Nov 03 11:16:02.945164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:02.945121    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" podStartSLOduration=2.945106843 podStartE2EDuration="2.945106843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:02.941175077 +0000 UTC m=+2793.146757129" watchObservedRunningTime="2026-11-03 11:16:02.945106843 +0000 UTC m=+2793.150688895"
Nov 03 11:16:03.016658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.016552    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743f219-4ae7-463e-956b-97e3ec89561f-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.016953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.016934    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743f219-4ae7-463e-956b-97e3ec89561f-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.017088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.017071    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bed6f31-3335-410e-a924-a345d610e597-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.017208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.017193    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.017339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.017326    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.017526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.017513    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.017711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.017695    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bed6f31-3335-410e-a924-a345d610e597-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.017969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.017946    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.018141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018038    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4xv\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-kube-api-access-5c4xv\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.018141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018103    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.018262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018179    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.018318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018281    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.018438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018420    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.018548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018528    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.019060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018660    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.019060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018720    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.019060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018909    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.019060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.018994    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngfc\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-kube-api-access-mngfc\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.019869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.019846    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.020240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.020219    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.020675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.020655    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.025887 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.020722    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743f219-4ae7-463e-956b-97e3ec89561f-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.026023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.021037    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.026113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.021550    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bed6f31-3335-410e-a924-a345d610e597-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.026237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.022280    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.026329 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.022408    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bed6f31-3335-410e-a924-a345d610e597-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.026421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.023761    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.026507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.025240    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.026644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.025486    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.028408 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.028366    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngfc\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-kube-api-access-mngfc\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.029559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.029332    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4xv\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-kube-api-access-5c4xv\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.032302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.032279    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.033131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.033089    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.033368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.033346    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.033436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.033376    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743f219-4ae7-463e-956b-97e3ec89561f-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-q64dp\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.034848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.034826    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.194124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.194090    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:03.203618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.196203    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:03.351923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.351887    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:03.360859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.360628    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:03.538009 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:16:03.537977    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1743f219_4ae7_463e_956b_97e3ec89561f.slice/crio-a9d1e7662a64fce535b4517f009427226e17385364879b1a2b1b2046208d5de4 WatchSource:0}: Error finding container a9d1e7662a64fce535b4517f009427226e17385364879b1a2b1b2046208d5de4: Status 404 returned error can't find the container with id a9d1e7662a64fce535b4517f009427226e17385364879b1a2b1b2046208d5de4
Nov 03 11:16:03.541881 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:16:03.541852    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bed6f31_3335_410e_a924_a345d610e597.slice/crio-0a54b503b8e961e9bf43c280223a7fb8dab371b4e5ee8e101fb5bfecc8050226 WatchSource:0}: Error finding container 0a54b503b8e961e9bf43c280223a7fb8dab371b4e5ee8e101fb5bfecc8050226: Status 404 returned error can't find the container with id 0a54b503b8e961e9bf43c280223a7fb8dab371b4e5ee8e101fb5bfecc8050226
Nov 03 11:16:03.542231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.542214    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"]
Nov 03 11:16:03.542301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.542212    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.542301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.542287    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.542420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.542331    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.544914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.544884    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.545031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.544956    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.545031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.545002    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.550165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.550140    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"]
Nov 03 11:16:03.673587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.673431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:03.942757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.942718    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl" event={"ID":"2b210b14-173e-44af-a9c1-e186d9572370","Type":"ContainerDied","Data":"54ca5f8d01f229c78eba31786752fea9bafa221b44102b63c0f6635f25dc9988"}
Nov 03 11:16:03.943681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.943653    5817 generic.go:356] "Generic (PLEG): container finished" podID="2b210b14-173e-44af-a9c1-e186d9572370" containerID="54ca5f8d01f229c78eba31786752fea9bafa221b44102b63c0f6635f25dc9988" exitCode=0
Nov 03 11:16:03.946348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.946323    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" event={"ID":"2bed6f31-3335-410e-a924-a345d610e597","Type":"ContainerStarted","Data":"5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215"}
Nov 03 11:16:03.946515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.946371    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" event={"ID":"2bed6f31-3335-410e-a924-a345d610e597","Type":"ContainerStarted","Data":"0a54b503b8e961e9bf43c280223a7fb8dab371b4e5ee8e101fb5bfecc8050226"}
Nov 03 11:16:03.948595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.948558    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" event={"ID":"1743f219-4ae7-463e-956b-97e3ec89561f","Type":"ContainerStarted","Data":"eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857"}
Nov 03 11:16:03.948768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.948748    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" event={"ID":"1743f219-4ae7-463e-956b-97e3ec89561f","Type":"ContainerStarted","Data":"a9d1e7662a64fce535b4517f009427226e17385364879b1a2b1b2046208d5de4"}
Nov 03 11:16:03.949587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.949521    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:03.951217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:03.951193    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:16:04.018596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.016444    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podStartSLOduration=2.016424044 podStartE2EDuration="2.016424044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:04.012133493 +0000 UTC m=+2794.217715547" watchObservedRunningTime="2026-11-03 11:16:04.016424044 +0000 UTC m=+2794.222006102"
Nov 03 11:16:04.059634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.057610    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podStartSLOduration=2.057565301 podStartE2EDuration="2.057565301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:04.053932629 +0000 UTC m=+2794.259514682" watchObservedRunningTime="2026-11-03 11:16:04.057565301 +0000 UTC m=+2794.263147350"
Nov 03 11:16:04.204084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.198665    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:04.204084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.198784    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:04.204084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.201210    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 11:16:04.204084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.201302    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 11:16:04.204084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.201675    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 03 11:16:04.204084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.201725    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 03 11:16:04.458751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:04.458557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:05.182751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.142429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:05.280124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.280021    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 03 11:16:05.280373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.280344    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 03 11:16:05.281023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.280996    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 11:16:05.281223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.281198    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 11:16:05.574895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.574821    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:05.712679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.701989    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-openshift-ca\") pod \"2b210b14-173e-44af-a9c1-e186d9572370\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") "
Nov 03 11:16:05.725880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.710999    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-openshift-ca" pod "2b210b14-173e-44af-a9c1-e186d9572370" (UID: "2b210b14-173e-44af-a9c1-e186d9572370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:05.743654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.727845    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b210b14-173e-44af-a9c1-e186d9572370-kube-api-access-js8d6\" (UniqueName: \"kubernetes.io/projected/2b210b14-173e-44af-a9c1-e186d9572370-kube-api-access-js8d6\") pod \"2b210b14-173e-44af-a9c1-e186d9572370\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") "
Nov 03 11:16:05.743654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.727942    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-xks-ca\") pod \"2b210b14-173e-44af-a9c1-e186d9572370\" (UID: \"2b210b14-173e-44af-a9c1-e186d9572370\") "
Nov 03 11:16:05.743654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.728875    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:05.763614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.762818    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b210b14-173e-44af-a9c1-e186d9572370-kube-api-access-js8d6" pod "2b210b14-173e-44af-a9c1-e186d9572370" (UID: "2b210b14-173e-44af-a9c1-e186d9572370"). InnerVolumeSpecName "kube-api-access-js8d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:05.873755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.840756    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-xks-ca" pod "2b210b14-173e-44af-a9c1-e186d9572370" (UID: "2b210b14-173e-44af-a9c1-e186d9572370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:16:05.934594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.926850    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8d6\" (UniqueName: \"kubernetes.io/projected/2b210b14-173e-44af-a9c1-e186d9572370-kube-api-access-js8d6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:05.934594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:05.926889    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b210b14-173e-44af-a9c1-e186d9572370-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:06.226161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.226128    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 11:16:06.226816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.226774    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 11:16:06.230593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.227287    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 03 11:16:06.230593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.227339    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 03 11:16:06.300976 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.300935    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl" event={"ID":"2b210b14-173e-44af-a9c1-e186d9572370","Type":"ContainerDied","Data":"ebb2dc7780582bb16a75fbeb47df89df4f6bc4217a048ee8d87a3556d5f82cd1"}
Nov 03 11:16:06.301240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.301220    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb2dc7780582bb16a75fbeb47df89df4f6bc4217a048ee8d87a3556d5f82cd1"
Nov 03 11:16:06.309602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.301397    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"
Nov 03 11:16:06.372008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.371964    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:06.998640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:06.995748    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:07.209596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:07.197126    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 11:16:07.209596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:07.197192    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 11:16:07.209596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:07.197606    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 03 11:16:07.209596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:07.197645    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 03 11:16:07.209596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:07.203169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:07.976550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:07.976346    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:08.147610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:08.147553    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:08.196000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:08.195960    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 11:16:08.196305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:08.196262    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 11:16:08.198160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:08.198134    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 03 11:16:08.198373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:08.198348    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 03 11:16:08.984732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:08.984692    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:09.353891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:09.336886    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 03 11:16:09.353891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:09.336964    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 03 11:16:09.353891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:09.337430    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 11:16:09.353891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:09.337470    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 11:16:10.000990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:10.000794    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:10.233599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:10.218911    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 03 11:16:10.233599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:10.218984    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 03 11:16:10.233599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:10.219387    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 11:16:10.233599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:10.219417    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 11:16:10.787600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:10.781355    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:11.195706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.195669    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 11:16:11.196179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.195735    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 11:16:11.197013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.196987    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 03 11:16:11.197074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.197051    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 03 11:16:11.359287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.359255    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"]
Nov 03 11:16:11.360421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.360398    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2b210b14-173e-44af-a9c1-e186d9572370" containerName="revoke-keys"
Nov 03 11:16:11.360543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.360532    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b210b14-173e-44af-a9c1-e186d9572370" containerName="revoke-keys"
Nov 03 11:16:11.360817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.360806    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b210b14-173e-44af-a9c1-e186d9572370" containerName="revoke-keys"
Nov 03 11:16:11.372673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.372644    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"]
Nov 03 11:16:11.372799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.372754    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.377874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.377851    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:16:11.377994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.377925    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:16:11.479673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.479545    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprt2\" (UniqueName: \"kubernetes.io/projected/af3009ca-a4e6-46f3-99fa-9da536815739-kube-api-access-sprt2\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.479827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.479704    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.479961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.479947    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.581206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.581173    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.581420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.581358    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprt2\" (UniqueName: \"kubernetes.io/projected/af3009ca-a4e6-46f3-99fa-9da536815739-kube-api-access-sprt2\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.581495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.581425    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.582024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.581998    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.583818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.583793    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.589114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.589090    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprt2\" (UniqueName: \"kubernetes.io/projected/af3009ca-a4e6-46f3-99fa-9da536815739-kube-api-access-sprt2\") pod \"maas-api-revoke-sub-bed0d818-4jljw\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.699587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.699549    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:11.782432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:11.782266    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:12.137405 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:16:12.137359    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3009ca_a4e6_46f3_99fa_9da536815739.slice/crio-be69cad971cfefb7fe8d5590948004b70c39fa34b8b7866981b7dedd3e42cf2f WatchSource:0}: Error finding container be69cad971cfefb7fe8d5590948004b70c39fa34b8b7866981b7dedd3e42cf2f: Status 404 returned error can't find the container with id be69cad971cfefb7fe8d5590948004b70c39fa34b8b7866981b7dedd3e42cf2f
Nov 03 11:16:12.138735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.138707    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"]
Nov 03 11:16:12.202546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.202521    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:12.203090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.203067    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:12.203177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.203116    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:12.203796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.203539    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:12.203914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.203895    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:16:12.204243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.204225    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:16:12.508957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.508929    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw" event={"ID":"af3009ca-a4e6-46f3-99fa-9da536815739","Type":"ContainerStarted","Data":"be69cad971cfefb7fe8d5590948004b70c39fa34b8b7866981b7dedd3e42cf2f"}
Nov 03 11:16:12.692598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:12.691631    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:13.466352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:13.466310    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:13.598168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:13.598121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:13.599435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:13.599404    5817 generic.go:356] "Generic (PLEG): container finished" podID="af3009ca-a4e6-46f3-99fa-9da536815739" containerID="83bd7978a916e6713bb5f30cc73efda4f2debf5d7f77fee1ef4fea58a6c1fa9b" exitCode=0
Nov 03 11:16:13.599758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:13.599734    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw" event={"ID":"af3009ca-a4e6-46f3-99fa-9da536815739","Type":"ContainerDied","Data":"83bd7978a916e6713bb5f30cc73efda4f2debf5d7f77fee1ef4fea58a6c1fa9b"}
Nov 03 11:16:13.731249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:13.731139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:14.849672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:14.839715    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:14.917989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:14.917918    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:14.999747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:14.999707    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3009ca-a4e6-46f3-99fa-9da536815739-kube-api-access-sprt2\" (UniqueName: \"kubernetes.io/projected/af3009ca-a4e6-46f3-99fa-9da536815739-kube-api-access-sprt2\") pod \"af3009ca-a4e6-46f3-99fa-9da536815739\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") "
Nov 03 11:16:14.999955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:14.999805    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-openshift-ca\") pod \"af3009ca-a4e6-46f3-99fa-9da536815739\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") "
Nov 03 11:16:14.999955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:14.999907    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-xks-ca\") pod \"af3009ca-a4e6-46f3-99fa-9da536815739\" (UID: \"af3009ca-a4e6-46f3-99fa-9da536815739\") "
Nov 03 11:16:15.000540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.000500    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-openshift-ca" pod "af3009ca-a4e6-46f3-99fa-9da536815739" (UID: "af3009ca-a4e6-46f3-99fa-9da536815739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:15.001179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.001153    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:15.004690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.004648    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3009ca-a4e6-46f3-99fa-9da536815739-kube-api-access-sprt2" pod "af3009ca-a4e6-46f3-99fa-9da536815739" (UID: "af3009ca-a4e6-46f3-99fa-9da536815739"). InnerVolumeSpecName "kube-api-access-sprt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:15.011288 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.011253    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-xks-ca" pod "af3009ca-a4e6-46f3-99fa-9da536815739" (UID: "af3009ca-a4e6-46f3-99fa-9da536815739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:16:15.032901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.032863    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"]
Nov 03 11:16:15.040345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.040321    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b19af0-7mcsb"]
Nov 03 11:16:15.102457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.102428    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprt2\" (UniqueName: \"kubernetes.io/projected/af3009ca-a4e6-46f3-99fa-9da536815739-kube-api-access-sprt2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:15.102457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.102458    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3009ca-a4e6-46f3-99fa-9da536815739-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:15.453460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.453424    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:15.715687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.699147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:15.732596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.732221    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw" event={"ID":"af3009ca-a4e6-46f3-99fa-9da536815739","Type":"ContainerDied","Data":"be69cad971cfefb7fe8d5590948004b70c39fa34b8b7866981b7dedd3e42cf2f"}
Nov 03 11:16:15.732596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.732315    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be69cad971cfefb7fe8d5590948004b70c39fa34b8b7866981b7dedd3e42cf2f"
Nov 03 11:16:15.732596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.732388    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"
Nov 03 11:16:15.857883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:15.857843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:16.044871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:16.044772    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9" path="/var/lib/kubelet/pods/e3c919e1-fde6-4f1c-a5cf-80fcc9712fa9/volumes"
Nov 03 11:16:16.756025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:16.755876    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:16.986642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:16.986603    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:17.237110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:17.237078    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:18.177603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:18.165786    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:18.406059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:18.406025    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:18.846242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:18.846207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:19.087733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:19.087697    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:19.850148 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:19.849802    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:20.586516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:20.586475    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:21.222737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:21.222701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:21.977611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:21.973598    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:22.417559 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:16:22.417477    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:16:22.705645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.705610    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"]
Nov 03 11:16:22.706647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.706617    5817 cpu_manager.go:414] "containerMap: removing container" podUID="af3009ca-a4e6-46f3-99fa-9da536815739" containerName="revoke-keys"
Nov 03 11:16:22.706647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.706641    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3009ca-a4e6-46f3-99fa-9da536815739" containerName="revoke-keys"
Nov 03 11:16:22.706903 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.706876    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3009ca-a4e6-46f3-99fa-9da536815739" containerName="revoke-keys"
Nov 03 11:16:22.716398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.716377    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:22.718751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.718733    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"]
Nov 03 11:16:22.724637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.724613    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:22.907187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.907146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:22.983646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.933095    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:22.984181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.984142    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g79vn\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-kube-api-access-g79vn\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:22.984435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.984416    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:22.984624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:22.984605    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.102647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.093565    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.106787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.106751    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.115650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.115616    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.122603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.116009    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.129612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.123111    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g79vn\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-kube-api-access-g79vn\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.138596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.122591    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.173665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.163199    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.173665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.163270    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"]
Nov 03 11:16:23.188702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.188397    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g79vn\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-kube-api-access-g79vn\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.203595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.200779    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"]
Nov 03 11:16:23.203595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.200909    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.265729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.239094    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.286640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.270080    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.308597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.287472    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.309329 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.309292    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvp9\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-kube-api-access-dvvp9\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.681661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.680653    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:23.711988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.711936    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.712602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.712542    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvp9\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-kube-api-access-dvvp9\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.713139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.713108    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.713599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.713560    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.819107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.819057    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.907977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.907915    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:16:23.977697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.977623    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:23.980589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:23.980023    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:24.006964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.006650    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvp9\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-kube-api-access-dvvp9\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:24.023335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.023310    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"]
Nov 03 11:16:24.030286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.030259    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"]
Nov 03 11:16:24.030451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.030365    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.034453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.034429    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:16:24.034453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.034437    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:16:24.185208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.185176    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:24.213250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.213216    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.213462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.213434    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvmqk\" (UniqueName: \"kubernetes.io/projected/4588367e-2641-4dc2-a783-99248c2de54c-kube-api-access-cvmqk\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.213803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.213781    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.323927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.315759    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.323927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.315890    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.323927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.316052    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvmqk\" (UniqueName: \"kubernetes.io/projected/4588367e-2641-4dc2-a783-99248c2de54c-kube-api-access-cvmqk\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.323927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.318694    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.390802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.380021    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:24.390802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.389010    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.393604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.393401    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"]
Nov 03 11:16:24.411271 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:16:24.409526    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4c140b_d39f_4f2d_beab_e84937b975cd.slice/crio-6a4d792e1132e25760cac39648072a03b553d451b6e6b087fbf205f86c7198dc WatchSource:0}: Error finding container 6a4d792e1132e25760cac39648072a03b553d451b6e6b087fbf205f86c7198dc: Status 404 returned error can't find the container with id 6a4d792e1132e25760cac39648072a03b553d451b6e6b087fbf205f86c7198dc
Nov 03 11:16:24.411271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.411258    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvmqk\" (UniqueName: \"kubernetes.io/projected/4588367e-2641-4dc2-a783-99248c2de54c-kube-api-access-cvmqk\") pod \"maas-api-revoke-sub-8f58ea23-rh677\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.657421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.657390    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:24.672160 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:16:24.672135    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42051072_0bc3_4c37_84f2_43a38eb07aec.slice/crio-2b2d84fb1c2a2536244800a61ce0127e637dc28e7b3164dcfccffc5735aa254e WatchSource:0}: Error finding container 2b2d84fb1c2a2536244800a61ce0127e637dc28e7b3164dcfccffc5735aa254e: Status 404 returned error can't find the container with id 2b2d84fb1c2a2536244800a61ce0127e637dc28e7b3164dcfccffc5735aa254e
Nov 03 11:16:24.674454 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.674431    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"]
Nov 03 11:16:24.973698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:24.971961    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:25.039888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.037990    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"]
Nov 03 11:16:25.058145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.058108    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d" event={"ID":"42051072-0bc3-4c37-84f2-43a38eb07aec","Type":"ContainerStarted","Data":"f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389"}
Nov 03 11:16:25.058304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.058174    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d" event={"ID":"42051072-0bc3-4c37-84f2-43a38eb07aec","Type":"ContainerStarted","Data":"2b2d84fb1c2a2536244800a61ce0127e637dc28e7b3164dcfccffc5735aa254e"}
Nov 03 11:16:25.058304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.058203    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:25.060955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.060926    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677" event={"ID":"4588367e-2641-4dc2-a783-99248c2de54c","Type":"ContainerStarted","Data":"ccc6fc0b6911a374f707fd147985a4181eb6438938c63d7d3996d54ebb0f78fa"}
Nov 03 11:16:25.063227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.063204    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d" event={"ID":"0a4c140b-d39f-4f2d-beab-e84937b975cd","Type":"ContainerStarted","Data":"48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908"}
Nov 03 11:16:25.063278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.063252    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d" event={"ID":"0a4c140b-d39f-4f2d-beab-e84937b975cd","Type":"ContainerStarted","Data":"6a4d792e1132e25760cac39648072a03b553d451b6e6b087fbf205f86c7198dc"}
Nov 03 11:16:25.063323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.063280    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:25.078196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.077961    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d" podStartSLOduration=2.077949149 podStartE2EDuration="2.077949149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:25.074830437 +0000 UTC m=+2815.280412489" watchObservedRunningTime="2026-11-03 11:16:25.077949149 +0000 UTC m=+2815.283531200"
Nov 03 11:16:25.093724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.093677    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d" podStartSLOduration=3.093662141 podStartE2EDuration="3.093662141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:25.091118779 +0000 UTC m=+2815.296700831" watchObservedRunningTime="2026-11-03 11:16:25.093662141 +0000 UTC m=+2815.299244188"
Nov 03 11:16:25.674653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:25.667445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:26.076140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:26.076100    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677" event={"ID":"4588367e-2641-4dc2-a783-99248c2de54c","Type":"ContainerStarted","Data":"9d5f1c3646198d0b3b6300021023d80653d800597d90a56cea2da3627149e31a"}
Nov 03 11:16:26.094446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:26.094391    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677" podStartSLOduration=3.094372763 podStartE2EDuration="3.094372763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:26.090586919 +0000 UTC m=+2816.296168962" watchObservedRunningTime="2026-11-03 11:16:26.094372763 +0000 UTC m=+2816.299954815"
Nov 03 11:16:26.271596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:26.265842    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:26.881805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:26.879536    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:27.088032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:27.087997    5817 generic.go:356] "Generic (PLEG): container finished" podID="4588367e-2641-4dc2-a783-99248c2de54c" containerID="9d5f1c3646198d0b3b6300021023d80653d800597d90a56cea2da3627149e31a" exitCode=0
Nov 03 11:16:27.088505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:27.088079    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677" event={"ID":"4588367e-2641-4dc2-a783-99248c2de54c","Type":"ContainerDied","Data":"9d5f1c3646198d0b3b6300021023d80653d800597d90a56cea2da3627149e31a"}
Nov 03 11:16:27.487606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:27.485914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:28.162600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.161779    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:28.391067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.391039    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:28.543344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.543310    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-openshift-ca\") pod \"4588367e-2641-4dc2-a783-99248c2de54c\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") "
Nov 03 11:16:28.543519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.543427    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4588367e-2641-4dc2-a783-99248c2de54c-kube-api-access-cvmqk\" (UniqueName: \"kubernetes.io/projected/4588367e-2641-4dc2-a783-99248c2de54c-kube-api-access-cvmqk\") pod \"4588367e-2641-4dc2-a783-99248c2de54c\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") "
Nov 03 11:16:28.543519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.543480    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-xks-ca\") pod \"4588367e-2641-4dc2-a783-99248c2de54c\" (UID: \"4588367e-2641-4dc2-a783-99248c2de54c\") "
Nov 03 11:16:28.543696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.543669    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-openshift-ca" pod "4588367e-2641-4dc2-a783-99248c2de54c" (UID: "4588367e-2641-4dc2-a783-99248c2de54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:28.544190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.544169    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:28.545773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.545749    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4588367e-2641-4dc2-a783-99248c2de54c-kube-api-access-cvmqk" pod "4588367e-2641-4dc2-a783-99248c2de54c" (UID: "4588367e-2641-4dc2-a783-99248c2de54c"). InnerVolumeSpecName "kube-api-access-cvmqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:28.545773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.545753    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-xks-ca" pod "4588367e-2641-4dc2-a783-99248c2de54c" (UID: "4588367e-2641-4dc2-a783-99248c2de54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:16:28.645271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.645208    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvmqk\" (UniqueName: \"kubernetes.io/projected/4588367e-2641-4dc2-a783-99248c2de54c-kube-api-access-cvmqk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:28.645271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.645233    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4588367e-2641-4dc2-a783-99248c2de54c-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:28.805283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:28.805242    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:29.112435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:29.112407    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"
Nov 03 11:16:29.112644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:29.112445    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677" event={"ID":"4588367e-2641-4dc2-a783-99248c2de54c","Type":"ContainerDied","Data":"ccc6fc0b6911a374f707fd147985a4181eb6438938c63d7d3996d54ebb0f78fa"}
Nov 03 11:16:29.112644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:29.112491    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccc6fc0b6911a374f707fd147985a4181eb6438938c63d7d3996d54ebb0f78fa"
Nov 03 11:16:29.425323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:29.425221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:30.079643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:30.073351    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:30.861215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:30.850345    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:31.448664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:31.446071    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:32.104599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:32.099412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:32.774603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:32.767597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:33.369602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:33.368519    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:34.058695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.058654    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:34.384017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.381491    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"]
Nov 03 11:16:34.384017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.383599    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4588367e-2641-4dc2-a783-99248c2de54c" containerName="revoke-keys"
Nov 03 11:16:34.384017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.383621    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4588367e-2641-4dc2-a783-99248c2de54c" containerName="revoke-keys"
Nov 03 11:16:34.384616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.384219    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4588367e-2641-4dc2-a783-99248c2de54c" containerName="revoke-keys"
Nov 03 11:16:34.400789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.400762    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"]
Nov 03 11:16:34.401148 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.401129    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.407731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.406439    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:16:34.408059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.408040    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:16:34.494930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.494899    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.495107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.495083    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.495225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.495211    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbtv\" (UniqueName: \"kubernetes.io/projected/d4ac2f9f-936f-4925-b796-03c133b405ec-kube-api-access-kfbtv\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.596948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.596914    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.597099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.597037    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.597168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.597152    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbtv\" (UniqueName: \"kubernetes.io/projected/d4ac2f9f-936f-4925-b796-03c133b405ec-kube-api-access-kfbtv\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.597816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.597735    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.599733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.599705    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.605921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.605895    5817 scope.go:122] "RemoveContainer" containerID="92e20b582c71079870486fe65350fa751dca6a42ea87cb88a1c9fbe186556dec"
Nov 03 11:16:34.606107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.606084    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbtv\" (UniqueName: \"kubernetes.io/projected/d4ac2f9f-936f-4925-b796-03c133b405ec-kube-api-access-kfbtv\") pod \"maas-api-revoke-sub-beeaacc7-g25kc\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.645126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.645104    5817 scope.go:122] "RemoveContainer" containerID="ad399cbfa530632cb25adf084e695be09ff4c4e351da7aea2622dfd4c23d19bb"
Nov 03 11:16:34.656964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.656946    5817 scope.go:122] "RemoveContainer" containerID="b5b2188d0c4cbc6c65e2d23600fd7b1eeb525897d8042f4bad6f068a9f1d5923"
Nov 03 11:16:34.722227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.722203    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:34.778946 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:34.778758    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:35.107615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:35.105139    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"]
Nov 03 11:16:35.235327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:35.235292    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc" event={"ID":"d4ac2f9f-936f-4925-b796-03c133b405ec","Type":"ContainerStarted","Data":"5951aa747b281d6739dc84384ee699923ec3174401e6887ce8648695e0dee535"}
Nov 03 11:16:35.382645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:35.380429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:36.069598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:36.065452    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:36.102945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:36.101293    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:36.104221 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:36.103987    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:36.188846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:36.188685    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"]
Nov 03 11:16:36.198217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:36.197733    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"]
Nov 03 11:16:36.249869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:36.249831    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc" event={"ID":"d4ac2f9f-936f-4925-b796-03c133b405ec","Type":"ContainerStarted","Data":"d843dd7ea15d6547ea8ffece88f8cfbad66de9b08150c71639b7c7985fbd0e98"}
Nov 03 11:16:36.266036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:36.265989    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc" podStartSLOduration=2.265975797 podStartE2EDuration="2.265975797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:36.264614457 +0000 UTC m=+2826.470196509" watchObservedRunningTime="2026-11-03 11:16:36.265975797 +0000 UTC m=+2826.471557848"
Nov 03 11:16:36.822136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:36.821506    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:37.261157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:37.261125    5817 generic.go:356] "Generic (PLEG): container finished" podID="d4ac2f9f-936f-4925-b796-03c133b405ec" containerID="d843dd7ea15d6547ea8ffece88f8cfbad66de9b08150c71639b7c7985fbd0e98" exitCode=0
Nov 03 11:16:37.261848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:37.261227    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc" event={"ID":"d4ac2f9f-936f-4925-b796-03c133b405ec","Type":"ContainerDied","Data":"d843dd7ea15d6547ea8ffece88f8cfbad66de9b08150c71639b7c7985fbd0e98"}
Nov 03 11:16:37.443076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:37.443038    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:38.075278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.074557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:38.459038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.459014    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:38.539127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.539093    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ac2f9f-936f-4925-b796-03c133b405ec-kube-api-access-kfbtv\" (UniqueName: \"kubernetes.io/projected/d4ac2f9f-936f-4925-b796-03c133b405ec-kube-api-access-kfbtv\") pod \"d4ac2f9f-936f-4925-b796-03c133b405ec\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") "
Nov 03 11:16:38.539330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.539310    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-xks-ca\") pod \"d4ac2f9f-936f-4925-b796-03c133b405ec\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") "
Nov 03 11:16:38.539483 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.539462    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-openshift-ca\") pod \"d4ac2f9f-936f-4925-b796-03c133b405ec\" (UID: \"d4ac2f9f-936f-4925-b796-03c133b405ec\") "
Nov 03 11:16:38.540031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.539995    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-openshift-ca" pod "d4ac2f9f-936f-4925-b796-03c133b405ec" (UID: "d4ac2f9f-936f-4925-b796-03c133b405ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:38.540552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.540530    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:38.546675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.546646    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ac2f9f-936f-4925-b796-03c133b405ec-kube-api-access-kfbtv" pod "d4ac2f9f-936f-4925-b796-03c133b405ec" (UID: "d4ac2f9f-936f-4925-b796-03c133b405ec"). InnerVolumeSpecName "kube-api-access-kfbtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:38.547682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.547657    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-xks-ca" pod "d4ac2f9f-936f-4925-b796-03c133b405ec" (UID: "d4ac2f9f-936f-4925-b796-03c133b405ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:16:38.642143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.642075    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ac2f9f-936f-4925-b796-03c133b405ec-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:38.642143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:38.642103    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbtv\" (UniqueName: \"kubernetes.io/projected/d4ac2f9f-936f-4925-b796-03c133b405ec-kube-api-access-kfbtv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:39.124933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:39.124837    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:39.338804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:39.338768    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:39.344386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:39.344306    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"
Nov 03 11:16:39.344386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:39.344338    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc" event={"ID":"d4ac2f9f-936f-4925-b796-03c133b405ec","Type":"ContainerDied","Data":"5951aa747b281d6739dc84384ee699923ec3174401e6887ce8648695e0dee535"}
Nov 03 11:16:39.344593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:39.344414    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5951aa747b281d6739dc84384ee699923ec3174401e6887ce8648695e0dee535"
Nov 03 11:16:40.403598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:40.396526    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:41.195599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:41.194675    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7" podUID="b24ebefd-9977-4a72-8103-9cacaf5d518c" containerName="payload-processing" containerID="cri-o://92fbda87e394a01efbc5757774021c6afc675e9c8115292ef605afde172bcce6" gracePeriod=25
Nov 03 11:16:41.208942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:41.208646    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll" podUID="7dafcc8a-f9aa-433f-a048-5033734fa494" containerName="payload-processing" containerID="cri-o://9a1706a5be60e20031d3a8621dcc5fc1b61d0248a05ea9500425ace076ff29d0" gracePeriod=25
Nov 03 11:16:41.643768 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:16:41.642177    5817 logging.go:55] [core] [Channel #11487 SubChannel #11488]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.37:50052", ServerName: "10.133.1.37:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.37:50052: connect: connection refused"
Nov 03 11:16:41.699930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:41.698016    5817 generic.go:356] "Generic (PLEG): container finished" podID="7dafcc8a-f9aa-433f-a048-5033734fa494" containerID="9a1706a5be60e20031d3a8621dcc5fc1b61d0248a05ea9500425ace076ff29d0" exitCode=0
Nov 03 11:16:41.699930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:41.698153    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll" event={"ID":"7dafcc8a-f9aa-433f-a048-5033734fa494","Type":"ContainerDied","Data":"9a1706a5be60e20031d3a8621dcc5fc1b61d0248a05ea9500425ace076ff29d0"}
Nov 03 11:16:41.699930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:41.699858    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:41.705366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:41.704869    5817 generic.go:356] "Generic (PLEG): container finished" podID="b24ebefd-9977-4a72-8103-9cacaf5d518c" containerID="92fbda87e394a01efbc5757774021c6afc675e9c8115292ef605afde172bcce6" exitCode=0
Nov 03 11:16:41.705366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:41.705281    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7" event={"ID":"b24ebefd-9977-4a72-8103-9cacaf5d518c","Type":"ContainerDied","Data":"92fbda87e394a01efbc5757774021c6afc675e9c8115292ef605afde172bcce6"}
Nov 03 11:16:41.949455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:41.948985    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:16:42.080659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.066290    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-plugins-config-volume\") pod \"b24ebefd-9977-4a72-8103-9cacaf5d518c\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") "
Nov 03 11:16:42.080659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.066464    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-routing-overlay-volume\") pod \"b24ebefd-9977-4a72-8103-9cacaf5d518c\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") "
Nov 03 11:16:42.080659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.066632    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-provider-credentials\") pod \"b24ebefd-9977-4a72-8103-9cacaf5d518c\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") "
Nov 03 11:16:42.080659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.066802    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-kube-api-access-664d6\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-kube-api-access-664d6\") pod \"b24ebefd-9977-4a72-8103-9cacaf5d518c\" (UID: \"b24ebefd-9977-4a72-8103-9cacaf5d518c\") "
Nov 03 11:16:42.178776 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.094913    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-routing-overlay-volume" pod "b24ebefd-9977-4a72-8103-9cacaf5d518c" (UID: "b24ebefd-9977-4a72-8103-9cacaf5d518c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:42.179237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.179202    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-plugins-config-volume" pod "b24ebefd-9977-4a72-8103-9cacaf5d518c" (UID: "b24ebefd-9977-4a72-8103-9cacaf5d518c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:42.213619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.187833    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:42.213619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.187870    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24ebefd-9977-4a72-8103-9cacaf5d518c-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:42.314597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.301934    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-provider-credentials" pod "b24ebefd-9977-4a72-8103-9cacaf5d518c" (UID: "b24ebefd-9977-4a72-8103-9cacaf5d518c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:42.336863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.336776    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-kube-api-access-664d6" pod "b24ebefd-9977-4a72-8103-9cacaf5d518c" (UID: "b24ebefd-9977-4a72-8103-9cacaf5d518c"). InnerVolumeSpecName "kube-api-access-664d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:42.406747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.406687    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:42.406747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.406724    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664d6\" (UniqueName: \"kubernetes.io/projected/b24ebefd-9977-4a72-8103-9cacaf5d518c-kube-api-access-664d6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:42.498930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.498886    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:16:42.632520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.632418    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll" podUID="7dafcc8a-f9aa-433f-a048-5033734fa494" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.37:50052\" within 1s: context deadline exceeded"
Nov 03 11:16:42.682634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.653984    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-routing-overlay-volume\") pod \"7dafcc8a-f9aa-433f-a048-5033734fa494\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") "
Nov 03 11:16:42.682634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.654144    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-kube-api-access-5n7hf\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-kube-api-access-5n7hf\") pod \"7dafcc8a-f9aa-433f-a048-5033734fa494\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") "
Nov 03 11:16:42.682634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.654324    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-plugins-config-volume\") pod \"7dafcc8a-f9aa-433f-a048-5033734fa494\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") "
Nov 03 11:16:42.682634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.654639    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-provider-credentials\") pod \"7dafcc8a-f9aa-433f-a048-5033734fa494\" (UID: \"7dafcc8a-f9aa-433f-a048-5033734fa494\") "
Nov 03 11:16:42.686368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.685006    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-routing-overlay-volume" pod "7dafcc8a-f9aa-433f-a048-5033734fa494" (UID: "7dafcc8a-f9aa-433f-a048-5033734fa494"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:42.733338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.733301    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-plugins-config-volume" pod "7dafcc8a-f9aa-433f-a048-5033734fa494" (UID: "7dafcc8a-f9aa-433f-a048-5033734fa494"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:42.770813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.769341    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-provider-credentials" pod "7dafcc8a-f9aa-433f-a048-5033734fa494" (UID: "7dafcc8a-f9aa-433f-a048-5033734fa494"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:42.789673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.789410    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:42.789673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.789442    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:42.789673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.789471    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dafcc8a-f9aa-433f-a048-5033734fa494-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:42.831635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.831212    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-kube-api-access-5n7hf" pod "7dafcc8a-f9aa-433f-a048-5033734fa494" (UID: "7dafcc8a-f9aa-433f-a048-5033734fa494"). InnerVolumeSpecName "kube-api-access-5n7hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:42.916717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.898185    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:42.916717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.910550    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n7hf\" (UniqueName: \"kubernetes.io/projected/7dafcc8a-f9aa-433f-a048-5033734fa494-kube-api-access-5n7hf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:42.995607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.969534    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll" event={"ID":"7dafcc8a-f9aa-433f-a048-5033734fa494","Type":"ContainerDied","Data":"4abe2ecde22335b69cb28cf1897c6290ceddd5cc494f0821113e208dd83ca680"}
Nov 03 11:16:42.995607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.969628    5817 scope.go:122] "RemoveContainer" containerID="9a1706a5be60e20031d3a8621dcc5fc1b61d0248a05ea9500425ace076ff29d0"
Nov 03 11:16:42.995607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.969814    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"
Nov 03 11:16:42.996189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.996160    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7" event={"ID":"b24ebefd-9977-4a72-8103-9cacaf5d518c","Type":"ContainerDied","Data":"040a625bfce01f6c7fbddb97c1840418a675efaefb71a8b7ba6b1a0ead562f1d"}
Nov 03 11:16:42.996367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:42.996318    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"
Nov 03 11:16:43.075360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.074200    5817 scope.go:122] "RemoveContainer" containerID="92fbda87e394a01efbc5757774021c6afc675e9c8115292ef605afde172bcce6"
Nov 03 11:16:43.169986 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.169901    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"]
Nov 03 11:16:43.175868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.175833    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-79dbzkgll"]
Nov 03 11:16:43.180978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.180958    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"]
Nov 03 11:16:43.185729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.185706    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-5b75dzhj7"]
Nov 03 11:16:43.542030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.541992    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:43.878771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.878672    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:43.935058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.935025    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dafcc8a-f9aa-433f-a048-5033734fa494" path="/var/lib/kubelet/pods/7dafcc8a-f9aa-433f-a048-5033734fa494/volumes"
Nov 03 11:16:43.936105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:43.936070    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24ebefd-9977-4a72-8103-9cacaf5d518c" path="/var/lib/kubelet/pods/b24ebefd-9977-4a72-8103-9cacaf5d518c/volumes"
Nov 03 11:16:44.784599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:44.779508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:45.317654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:45.317612    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:46.068343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:46.068304    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:46.131859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:46.131361    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"]
Nov 03 11:16:46.235403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:46.235372    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:46.386746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:46.386658    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:46.434430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:46.434396    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"]
Nov 03 11:16:46.729611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:46.722786    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:46.918597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:46.901311    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:47.916174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:47.916107    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:48.057949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:48.057906    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"]
Nov 03 11:16:48.064803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:48.064774    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1c3efd-c8lvb"]
Nov 03 11:16:48.828593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:48.815548    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:49.563074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:49.563039    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:49.909612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:49.909486    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce303c9a-dd89-4538-bebd-cda0494cf78e" path="/var/lib/kubelet/pods/ce303c9a-dd89-4538-bebd-cda0494cf78e/volumes"
Nov 03 11:16:50.318617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:50.313203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:51.123339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.122259    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:51.132688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.132643    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d" podUID="0a4c140b-d39f-4f2d-beab-e84937b975cd" containerName="payload-processing" containerID="cri-o://48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908" gracePeriod=25
Nov 03 11:16:51.435709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.435668    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d" podUID="42051072-0bc3-4c37-84f2-43a38eb07aec" containerName="payload-processing" containerID="cri-o://f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389" gracePeriod=25
Nov 03 11:16:51.539619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.539548    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:51.658871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.658838    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-plugins-config-volume\") pod \"0a4c140b-d39f-4f2d-beab-e84937b975cd\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") "
Nov 03 11:16:51.659064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.659045    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-provider-credentials\") pod \"0a4c140b-d39f-4f2d-beab-e84937b975cd\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") "
Nov 03 11:16:51.659189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.659170    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-kube-api-access-g79vn\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-kube-api-access-g79vn\") pod \"0a4c140b-d39f-4f2d-beab-e84937b975cd\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") "
Nov 03 11:16:51.659342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.659313    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-plugins-config-volume" pod "0a4c140b-d39f-4f2d-beab-e84937b975cd" (UID: "0a4c140b-d39f-4f2d-beab-e84937b975cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:51.659342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.659330    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-routing-overlay-volume\") pod \"0a4c140b-d39f-4f2d-beab-e84937b975cd\" (UID: \"0a4c140b-d39f-4f2d-beab-e84937b975cd\") "
Nov 03 11:16:51.659889 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.659860    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-routing-overlay-volume" pod "0a4c140b-d39f-4f2d-beab-e84937b975cd" (UID: "0a4c140b-d39f-4f2d-beab-e84937b975cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:51.660318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.660293    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:51.660395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.660325    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a4c140b-d39f-4f2d-beab-e84937b975cd-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:51.662478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.662451    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-kube-api-access-g79vn" pod "0a4c140b-d39f-4f2d-beab-e84937b975cd" (UID: "0a4c140b-d39f-4f2d-beab-e84937b975cd"). InnerVolumeSpecName "kube-api-access-g79vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:51.663071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.663047    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-provider-credentials" pod "0a4c140b-d39f-4f2d-beab-e84937b975cd" (UID: "0a4c140b-d39f-4f2d-beab-e84937b975cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:51.752735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.748855    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:51.766014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.762468    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:51.766014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.762502    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g79vn\" (UniqueName: \"kubernetes.io/projected/0a4c140b-d39f-4f2d-beab-e84937b975cd-kube-api-access-g79vn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:51.865150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.864792    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:51.966478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.966403    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-plugins-config-volume\") pod \"42051072-0bc3-4c37-84f2-43a38eb07aec\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") "
Nov 03 11:16:51.966739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.966717    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-provider-credentials\") pod \"42051072-0bc3-4c37-84f2-43a38eb07aec\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") "
Nov 03 11:16:51.966849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.966815    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-plugins-config-volume" pod "42051072-0bc3-4c37-84f2-43a38eb07aec" (UID: "42051072-0bc3-4c37-84f2-43a38eb07aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:51.966918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.966843    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-routing-overlay-volume\") pod \"42051072-0bc3-4c37-84f2-43a38eb07aec\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") "
Nov 03 11:16:51.966978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.966909    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-kube-api-access-dvvp9\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-kube-api-access-dvvp9\") pod \"42051072-0bc3-4c37-84f2-43a38eb07aec\" (UID: \"42051072-0bc3-4c37-84f2-43a38eb07aec\") "
Nov 03 11:16:51.967188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.967160    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-routing-overlay-volume" pod "42051072-0bc3-4c37-84f2-43a38eb07aec" (UID: "42051072-0bc3-4c37-84f2-43a38eb07aec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:16:51.967932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.967909    5817 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-routing-overlay-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:51.968047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.967936    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42051072-0bc3-4c37-84f2-43a38eb07aec-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:51.969279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.969260    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-provider-credentials" pod "42051072-0bc3-4c37-84f2-43a38eb07aec" (UID: "42051072-0bc3-4c37-84f2-43a38eb07aec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:51.969379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:51.969353    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-kube-api-access-dvvp9" pod "42051072-0bc3-4c37-84f2-43a38eb07aec" (UID: "42051072-0bc3-4c37-84f2-43a38eb07aec"). InnerVolumeSpecName "kube-api-access-dvvp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:16:52.069473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.069441    5817 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-provider-credentials\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:52.069473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.069472    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvvp9\" (UniqueName: \"kubernetes.io/projected/42051072-0bc3-4c37-84f2-43a38eb07aec-kube-api-access-dvvp9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:16:52.219645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.218780    5817 generic.go:356] "Generic (PLEG): container finished" podID="0a4c140b-d39f-4f2d-beab-e84937b975cd" containerID="48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908" exitCode=0
Nov 03 11:16:52.219645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.218822    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d" event={"ID":"0a4c140b-d39f-4f2d-beab-e84937b975cd","Type":"ContainerDied","Data":"48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908"}
Nov 03 11:16:52.219645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.218838    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"
Nov 03 11:16:52.219645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.218983    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d" event={"ID":"0a4c140b-d39f-4f2d-beab-e84937b975cd","Type":"ContainerDied","Data":"6a4d792e1132e25760cac39648072a03b553d451b6e6b087fbf205f86c7198dc"}
Nov 03 11:16:52.219645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.219014    5817 scope.go:122] "RemoveContainer" containerID="48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908"
Nov 03 11:16:52.221942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.221335    5817 generic.go:356] "Generic (PLEG): container finished" podID="42051072-0bc3-4c37-84f2-43a38eb07aec" containerID="f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389" exitCode=0
Nov 03 11:16:52.221942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.221431    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"
Nov 03 11:16:52.221942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.221729    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d" event={"ID":"42051072-0bc3-4c37-84f2-43a38eb07aec","Type":"ContainerDied","Data":"f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389"}
Nov 03 11:16:52.221942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.221786    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d" event={"ID":"42051072-0bc3-4c37-84f2-43a38eb07aec","Type":"ContainerDied","Data":"2b2d84fb1c2a2536244800a61ce0127e637dc28e7b3164dcfccffc5735aa254e"}
Nov 03 11:16:52.237080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.237056    5817 scope.go:122] "RemoveContainer" containerID="48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908"
Nov 03 11:16:52.237417 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:16:52.237398    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908\": container with ID starting with 48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908 not found: ID does not exist" containerID="48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908"
Nov 03 11:16:52.237618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.237424    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908"} err="failed to get container status \"48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908\": rpc error: code = NotFound desc = could not find container \"48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908\": container with ID starting with 48d4b98f9bc0f712197c31b5846fe1fb7636475a8b3fd0e7c63be5544f858908 not found: ID does not exist"
Nov 03 11:16:52.237618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.237440    5817 scope.go:122] "RemoveContainer" containerID="f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389"
Nov 03 11:16:52.243369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.243337    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"]
Nov 03 11:16:52.249848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.249824    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d8hm95d"]
Nov 03 11:16:52.250631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.250561    5817 scope.go:122] "RemoveContainer" containerID="f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389"
Nov 03 11:16:52.250963 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:16:52.250944    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389\": container with ID starting with f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389 not found: ID does not exist" containerID="f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389"
Nov 03 11:16:52.251021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.250973    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389"} err="failed to get container status \"f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389\": rpc error: code = NotFound desc = could not find container \"f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389\": container with ID starting with f9d2a118de45fffae7c9ccf8a3b445f5a1fee47d64f9dda146d1060332568389 not found: ID does not exist"
Nov 03 11:16:52.261302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.261283    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"]
Nov 03 11:16:52.266361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.266340    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5cbfm2q9d"]
Nov 03 11:16:52.446040 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:16:52.446002    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:16:52.578645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.573474    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:52.773708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.773674    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:52.908368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:52.908286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:53.247228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.247169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:53.388543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.388499    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:53.575066 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.574993    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"]
Nov 03 11:16:53.575973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.575941    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7dafcc8a-f9aa-433f-a048-5033734fa494" containerName="payload-processing"
Nov 03 11:16:53.575973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.575963    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dafcc8a-f9aa-433f-a048-5033734fa494" containerName="payload-processing"
Nov 03 11:16:53.576190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576036    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b24ebefd-9977-4a72-8103-9cacaf5d518c" containerName="payload-processing"
Nov 03 11:16:53.576190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576047    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24ebefd-9977-4a72-8103-9cacaf5d518c" containerName="payload-processing"
Nov 03 11:16:53.576190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576085    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0a4c140b-d39f-4f2d-beab-e84937b975cd" containerName="payload-processing"
Nov 03 11:16:53.576190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576092    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4c140b-d39f-4f2d-beab-e84937b975cd" containerName="payload-processing"
Nov 03 11:16:53.576190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576111    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d4ac2f9f-936f-4925-b796-03c133b405ec" containerName="revoke-keys"
Nov 03 11:16:53.576190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576118    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ac2f9f-936f-4925-b796-03c133b405ec" containerName="revoke-keys"
Nov 03 11:16:53.576190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576141    5817 cpu_manager.go:414] "containerMap: removing container" podUID="42051072-0bc3-4c37-84f2-43a38eb07aec" containerName="payload-processing"
Nov 03 11:16:53.576190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576149    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42051072-0bc3-4c37-84f2-43a38eb07aec" containerName="payload-processing"
Nov 03 11:16:53.576687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576347    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="42051072-0bc3-4c37-84f2-43a38eb07aec" containerName="payload-processing"
Nov 03 11:16:53.576687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576364    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dafcc8a-f9aa-433f-a048-5033734fa494" containerName="payload-processing"
Nov 03 11:16:53.576687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576386    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24ebefd-9977-4a72-8103-9cacaf5d518c" containerName="payload-processing"
Nov 03 11:16:53.576687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576402    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4c140b-d39f-4f2d-beab-e84937b975cd" containerName="payload-processing"
Nov 03 11:16:53.576687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.576416    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ac2f9f-936f-4925-b796-03c133b405ec" containerName="revoke-keys"
Nov 03 11:16:53.586755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.582131    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.598336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.598164    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b13404-openshift-default-dockercfg-bxcqw\""
Nov 03 11:16:53.608218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.608188    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"]
Nov 03 11:16:53.669380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.663982    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.725287    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.725471    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99eda1c3-5954-45d0-b159-d1f95545bd75-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.725504    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.725539    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.725968    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.726041    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpkz\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-kube-api-access-qnpkz\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.726103    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.726151    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.726225    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.810470    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.827827    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.827959    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99eda1c3-5954-45d0-b159-d1f95545bd75-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.828004    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.828040    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.828279    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.828324    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpkz\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-kube-api-access-qnpkz\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.828365    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.828416    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.828479    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.828961    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.844397    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.845045    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99eda1c3-5954-45d0-b159-d1f95545bd75-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.845548    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.862658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.845665    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.875610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.865004    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.915830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.900543    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpkz\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-kube-api-access-qnpkz\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.986600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.955950    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:53.986600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:53.984295    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:54.034125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.034034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:54.137109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.137018    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4c140b-d39f-4f2d-beab-e84937b975cd" path="/var/lib/kubelet/pods/0a4c140b-d39f-4f2d-beab-e84937b975cd/volumes"
Nov 03 11:16:54.137843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.137815    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42051072-0bc3-4c37-84f2-43a38eb07aec" path="/var/lib/kubelet/pods/42051072-0bc3-4c37-84f2-43a38eb07aec/volumes"
Nov 03 11:16:54.229320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.229286    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:54.425748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.377881    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:54.562611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.558709    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"]
Nov 03 11:16:54.562611 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:16:54.559384    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99eda1c3_5954_45d0_b159_d1f95545bd75.slice/crio-fbe4764091b43c44cdb6b94f2efa59f3f306297b1da45f6ba895ae263ce3e47d WatchSource:0}: Error finding container fbe4764091b43c44cdb6b94f2efa59f3f306297b1da45f6ba895ae263ce3e47d: Status 404 returned error can't find the container with id fbe4764091b43c44cdb6b94f2efa59f3f306297b1da45f6ba895ae263ce3e47d
Nov 03 11:16:54.564592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.563747    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:54.564592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.563835    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:54.564592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:54.563879    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:55.192878 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.192838    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:55.292461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.292425    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" event={"ID":"99eda1c3-5954-45d0-b159-d1f95545bd75","Type":"ContainerStarted","Data":"98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05"}
Nov 03 11:16:55.292712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.292483    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" event={"ID":"99eda1c3-5954-45d0-b159-d1f95545bd75","Type":"ContainerStarted","Data":"fbe4764091b43c44cdb6b94f2efa59f3f306297b1da45f6ba895ae263ce3e47d"}
Nov 03 11:16:55.294859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.294834    5817 generic.go:356] "Generic (PLEG): container finished" podID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerID="e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786" exitCode=0
Nov 03 11:16:55.294980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.294917    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" event={"ID":"dea678e2-cfc3-4ff5-8889-601be2dd2909","Type":"ContainerDied","Data":"e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786"}
Nov 03 11:16:55.295391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.295373    5817 scope.go:122] "RemoveContainer" containerID="e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786"
Nov 03 11:16:55.297714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.297687    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:55.297839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.297777    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:55.297911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.297840    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:55.319983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:55.319912    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" podStartSLOduration=2.319895469 podStartE2EDuration="2.319895469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:55.312192266 +0000 UTC m=+2845.517774317" watchObservedRunningTime="2026-11-03 11:16:55.319895469 +0000 UTC m=+2845.525477560"
Nov 03 11:16:56.215596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.211312    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:56.235608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.235236    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:56.252047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.252011    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:56.334133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.334060    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" event={"ID":"dea678e2-cfc3-4ff5-8889-601be2dd2909","Type":"ContainerStarted","Data":"731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976"}
Nov 03 11:16:56.334133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.334132    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:56.335822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.335804    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:16:56.776420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.776392    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:16:56.777470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.777446    5817 patch_prober.go:29] interesting pod/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:16:56.777612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:56.777502    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 11:16:57.168501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:57.168415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:57.717150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:57.717116    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:57.775847 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:57.775806    5817 patch_prober.go:29] interesting pod/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:16:57.776032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:57.775879    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 11:16:57.862390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:57.862343    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:58.320601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.316799    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:58.777289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.777252    5817 patch_prober.go:29] interesting pod/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:16:58.777970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.777328    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 11:16:58.943001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.942967    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:58.969700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.969673    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"]
Nov 03 11:16:58.977278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.977239    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"]
Nov 03 11:16:58.977454 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.977384    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:16:58.981452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.981422    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"]
Nov 03 11:16:58.982129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.982105    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b13404-dockercfg-b74lz\""
Nov 03 11:16:58.982280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:58.982191    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b13404\""
Nov 03 11:16:59.008782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.006266    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:16:59.111660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.107642    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"]
Nov 03 11:16:59.111660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.110370    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk74t\" (UniqueName: \"kubernetes.io/projected/838aae41-6959-4c5d-b1a8-348ca9375a72-kube-api-access-vk74t\") pod \"payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s\" (UID: \"838aae41-6959-4c5d-b1a8-348ca9375a72\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:16:59.111660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.110724    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838aae41-6959-4c5d-b1a8-348ca9375a72-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s\" (UID: \"838aae41-6959-4c5d-b1a8-348ca9375a72\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:16:59.111660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.110766    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d55c2c-2972-4fcf-96ca-0629f294a1b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b13404-766b656b97-mngzq\" (UID: \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:16:59.154603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.147911    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:59.266596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.241532    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdtd\" (UniqueName: \"kubernetes.io/projected/43d55c2c-2972-4fcf-96ca-0629f294a1b1-kube-api-access-5kdtd\") pod \"payload-processing-e2e-inf-a-b13404-766b656b97-mngzq\" (UID: \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:16:59.375647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.375068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:16:59.375647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.375166    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk74t\" (UniqueName: \"kubernetes.io/projected/838aae41-6959-4c5d-b1a8-348ca9375a72-kube-api-access-vk74t\") pod \"payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s\" (UID: \"838aae41-6959-4c5d-b1a8-348ca9375a72\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:16:59.375647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.375492    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838aae41-6959-4c5d-b1a8-348ca9375a72-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s\" (UID: \"838aae41-6959-4c5d-b1a8-348ca9375a72\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:16:59.375647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.375525    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d55c2c-2972-4fcf-96ca-0629f294a1b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b13404-766b656b97-mngzq\" (UID: \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:16:59.375647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.375618    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdtd\" (UniqueName: \"kubernetes.io/projected/43d55c2c-2972-4fcf-96ca-0629f294a1b1-kube-api-access-5kdtd\") pod \"payload-processing-e2e-inf-a-b13404-766b656b97-mngzq\" (UID: \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:16:59.407600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.377166    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838aae41-6959-4c5d-b1a8-348ca9375a72-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s\" (UID: \"838aae41-6959-4c5d-b1a8-348ca9375a72\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:16:59.431780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.428981    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d55c2c-2972-4fcf-96ca-0629f294a1b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b13404-766b656b97-mngzq\" (UID: \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:16:59.507598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.483463    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdtd\" (UniqueName: \"kubernetes.io/projected/43d55c2c-2972-4fcf-96ca-0629f294a1b1-kube-api-access-5kdtd\") pod \"payload-processing-e2e-inf-a-b13404-766b656b97-mngzq\" (UID: \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:16:59.535599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.519291    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk74t\" (UniqueName: \"kubernetes.io/projected/838aae41-6959-4c5d-b1a8-348ca9375a72-kube-api-access-vk74t\") pod \"payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s\" (UID: \"838aae41-6959-4c5d-b1a8-348ca9375a72\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:16:59.665525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.665439    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:16:59.797600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.771649    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:16:59.910982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.893172    5817 patch_prober.go:29] interesting pod/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:16:59.910982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.893232    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 11:16:59.920666 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:16:59.918563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:00.483375 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:17:00.483342    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838aae41_6959_4c5d_b1a8_348ca9375a72.slice/crio-858220e75dc7a4acb47f643316ea74861cd9d2b2cfcec32ecffdb804abfea3e7 WatchSource:0}: Error finding container 858220e75dc7a4acb47f643316ea74861cd9d2b2cfcec32ecffdb804abfea3e7: Status 404 returned error can't find the container with id 858220e75dc7a4acb47f643316ea74861cd9d2b2cfcec32ecffdb804abfea3e7
Nov 03 11:17:00.484317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:00.484292    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"]
Nov 03 11:17:00.538900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:00.538868    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"]
Nov 03 11:17:00.583475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:00.583447    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq" event={"ID":"43d55c2c-2972-4fcf-96ca-0629f294a1b1","Type":"ContainerStarted","Data":"9ae201c2935808867cac7111b0bede26fadf70a92eb4a84f7462c198b4df9044"}
Nov 03 11:17:00.585205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:00.585179    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s" event={"ID":"838aae41-6959-4c5d-b1a8-348ca9375a72","Type":"ContainerStarted","Data":"858220e75dc7a4acb47f643316ea74861cd9d2b2cfcec32ecffdb804abfea3e7"}
Nov 03 11:17:00.717604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:00.697802    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:17:00.717604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:00.701283    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:00.797620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:00.784713    5817 patch_prober.go:29] interesting pod/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:17:00.797620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:00.784770    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 11:17:01.561598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.557862    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:01.747765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.747730    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq" event={"ID":"43d55c2c-2972-4fcf-96ca-0629f294a1b1","Type":"ContainerStarted","Data":"d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6"}
Nov 03 11:17:01.748105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.748088    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:17:01.754385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.754356    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s" event={"ID":"838aae41-6959-4c5d-b1a8-348ca9375a72","Type":"ContainerStarted","Data":"04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5"}
Nov 03 11:17:01.755034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.755013    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:17:01.767789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.766813    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq" podStartSLOduration=3.766796456 podStartE2EDuration="3.766796456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:01.765680246 +0000 UTC m=+2851.971262309" watchObservedRunningTime="2026-11-03 11:17:01.766796456 +0000 UTC m=+2851.972378508"
Nov 03 11:17:01.775809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.775786    5817 patch_prober.go:29] interesting pod/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:17:01.775918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.775825    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 11:17:01.819655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:01.815025    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s" podStartSLOduration=3.815005717 podStartE2EDuration="3.815005717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:01.784371486 +0000 UTC m=+2851.989953536" watchObservedRunningTime="2026-11-03 11:17:01.815005717 +0000 UTC m=+2852.020587769"
Nov 03 11:17:02.365170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:02.365135    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:02.776592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:02.776536    5817 patch_prober.go:29] interesting pod/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:17:02.776801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:02.776604    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 11:17:03.189769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:03.189677    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:03.782351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:03.782323    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:17:03.783398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:03.783376    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:17:03.930109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:03.930065    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:04.574399 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:04.571715    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:05.336610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:05.336430    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:05.955596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:05.947681    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:06.687599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:06.685127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:07.384535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:07.384480    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:07.980599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:07.979132    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:08.160735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:08.160702    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:08.754069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:08.754030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:09.400260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:09.400217    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:09.958221 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:09.958184    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:10.138176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:10.138140    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:10.302265 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:10.302170    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:11.160598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:11.160293    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:11.503813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:11.503780    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:11.958325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:11.958286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:12.104257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.104224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:12.319973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.319879    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"]
Nov 03 11:17:12.335703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.335673    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.339140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.339114    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"]
Nov 03 11:17:12.340166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.340143    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c0a748-openshift-default-dockercfg-q6b72\""
Nov 03 11:17:12.405923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.405882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.472614    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5308b6-3a71-4bca-9b07-fe824d145610-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.472810    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.472853    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflzv\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-kube-api-access-bflzv\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.472906    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.472972    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.473077    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.473260    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.473314    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.474617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.473425    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.559475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.559257    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:12.591854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.591767    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.593446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.592143    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.593446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.592254    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5308b6-3a71-4bca-9b07-fe824d145610-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.593446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.592420    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.593446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.592453    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflzv\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-kube-api-access-bflzv\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.593446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.592505    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.593446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.592564    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.593446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.592672    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.593446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.592891    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.594104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.594065    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.594618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.594597    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.596004 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.595963    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5308b6-3a71-4bca-9b07-fe824d145610-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.597591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.597491    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.616607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.598077    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.620605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.620214    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.621374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.621338    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.629320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.629272    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflzv\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-kube-api-access-bflzv\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.641456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.641393    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.656607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.656457    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:12.826904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:12.826865    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:17:13.353280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:13.353038    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:13.354446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:13.354419    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"]
Nov 03 11:17:13.378355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:13.378298    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:13.378460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:13.378393    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:13.378460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:13.378425    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:13.781337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:13.781310    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:17:14.052290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:14.052208    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:14.158926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:14.158888    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" event={"ID":"8a5308b6-3a71-4bca-9b07-fe824d145610","Type":"ContainerStarted","Data":"204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec"}
Nov 03 11:17:14.159080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:14.158945    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" event={"ID":"8a5308b6-3a71-4bca-9b07-fe824d145610","Type":"ContainerStarted","Data":"f53fac2db11aafa275eba72b4ee44c91d75b980a52f493b51147f8cd677023a3"}
Nov 03 11:17:14.181885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:14.181839    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podStartSLOduration=2.181825576 podStartE2EDuration="2.181825576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:14.177547618 +0000 UTC m=+2864.383129668" watchObservedRunningTime="2026-11-03 11:17:14.181825576 +0000 UTC m=+2864.387407628"
Nov 03 11:17:14.658869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:14.658831    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:14.665427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:14.665405    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:14.754607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:14.754056    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:15.168663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:15.168614    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:15.169549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:15.169532    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:17:15.469774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:15.462163    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:16.480939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:16.480905    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:16.689118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:16.689088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:17.764376 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.754218    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:17.874304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.874271    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"]
Nov 03 11:17:17.880968 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.880945    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:17.882178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.882156    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"]
Nov 03 11:17:17.885918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.885899    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c0a748\""
Nov 03 11:17:17.887913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.887890    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c0a748-dockercfg-bxbb2\""
Nov 03 11:17:17.919811 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.919782    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"]
Nov 03 11:17:17.928507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.928477    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:17.929682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.929651    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"]
Nov 03 11:17:17.944834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.943627    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0f4d83-bc3f-4c99-9173-1efcb60086de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx\" (UID: \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:17.944834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:17.943710    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-775cm\" (UniqueName: \"kubernetes.io/projected/ad0f4d83-bc3f-4c99-9173-1efcb60086de-kube-api-access-775cm\") pod \"payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx\" (UID: \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:18.056110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.056010    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0f4d83-bc3f-4c99-9173-1efcb60086de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx\" (UID: \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:18.056110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.056080    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-775cm\" (UniqueName: \"kubernetes.io/projected/ad0f4d83-bc3f-4c99-9173-1efcb60086de-kube-api-access-775cm\") pod \"payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx\" (UID: \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:18.056358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.056173    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tqj\" (UniqueName: \"kubernetes.io/projected/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-kube-api-access-t9tqj\") pod \"payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts\" (UID: \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:18.056358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.056205    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts\" (UID: \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:18.075659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.057399    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0f4d83-bc3f-4c99-9173-1efcb60086de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx\" (UID: \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:18.095688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.079169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:18.171613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.169330    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tqj\" (UniqueName: \"kubernetes.io/projected/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-kube-api-access-t9tqj\") pod \"payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts\" (UID: \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:18.171613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.169389    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts\" (UID: \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:18.262630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.194826    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts\" (UID: \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:18.262630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.195150    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-775cm\" (UniqueName: \"kubernetes.io/projected/ad0f4d83-bc3f-4c99-9173-1efcb60086de-kube-api-access-775cm\") pod \"payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx\" (UID: \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:18.378384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.377048    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:18.439187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.435146    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tqj\" (UniqueName: \"kubernetes.io/projected/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-kube-api-access-t9tqj\") pod \"payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts\" (UID: \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:18.503795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.500237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:18.503795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.502347    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:18.781479 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:17:18.781454    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0f4d83_bc3f_4c99_9173_1efcb60086de.slice/crio-85bcf4259349dc98f2d14955de9a9923a1aa917a99bfdf3cd44010267551feab WatchSource:0}: Error finding container 85bcf4259349dc98f2d14955de9a9923a1aa917a99bfdf3cd44010267551feab: Status 404 returned error can't find the container with id 85bcf4259349dc98f2d14955de9a9923a1aa917a99bfdf3cd44010267551feab
Nov 03 11:17:18.782385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.782363    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"]
Nov 03 11:17:18.889144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.878119    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:18.912162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:18.910761    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"]
Nov 03 11:17:19.417038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.416998    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" event={"ID":"ad0f4d83-bc3f-4c99-9173-1efcb60086de","Type":"ContainerStarted","Data":"b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769"}
Nov 03 11:17:19.417291 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.417097    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" event={"ID":"ad0f4d83-bc3f-4c99-9173-1efcb60086de","Type":"ContainerStarted","Data":"85bcf4259349dc98f2d14955de9a9923a1aa917a99bfdf3cd44010267551feab"}
Nov 03 11:17:19.417291 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.417249    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:19.419185 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.419158    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts" event={"ID":"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82","Type":"ContainerStarted","Data":"f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903"}
Nov 03 11:17:19.419349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.419201    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts" event={"ID":"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82","Type":"ContainerStarted","Data":"507d1815c8f1853a47e5c1bbd7940cd039b8bd28a1833c828322ebce1f52334e"}
Nov 03 11:17:19.419349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.419220    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:19.434992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.434937    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" podStartSLOduration=2.434921088 podStartE2EDuration="2.434921088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:19.432795087 +0000 UTC m=+2869.638377138" watchObservedRunningTime="2026-11-03 11:17:19.434921088 +0000 UTC m=+2869.640503141"
Nov 03 11:17:19.456715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.456672    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts" podStartSLOduration=2.456658831 podStartE2EDuration="2.456658831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:19.450080488 +0000 UTC m=+2869.655662539" watchObservedRunningTime="2026-11-03 11:17:19.456658831 +0000 UTC m=+2869.662240882"
Nov 03 11:17:19.848597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:19.844635    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:20.542768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:20.542730    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:21.370601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:21.365539    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:22.174123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:22.173064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:22.415598 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:17:22.415108    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:17:22.945604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:22.944436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:23.657604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:23.638483    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:24.409602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:24.402119    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:25.162465 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:25.162432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:25.296174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:25.296141    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:25.997370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:25.997332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:26.802800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:26.801229    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:27.497261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:27.497226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:28.243954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:28.243920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:28.901449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:28.901412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:29.103913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:29.103878    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:17:30.017301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:30.017261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:30.473121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:30.473089    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:17:30.474917 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:30.474892    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:17:31.396605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:31.396530    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:32.620370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:32.620334    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:32.697737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:32.697668    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:17:32.699195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:32.699163    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:17:32.708688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:32.708652    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:17:32.874677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:32.873182    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:17:32.884524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:32.880444    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:17:32.884846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:32.884812    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:17:33.394600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:33.391127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:34.076285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:34.076245    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:34.735068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:34.735037    5817 scope.go:122] "RemoveContainer" containerID="0a6d32544b229637c148875a1edcbd9b24816bc58d000dc581910733244027af"
Nov 03 11:17:34.950596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:34.949463    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:35.707473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:35.703061    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:36.285598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:36.281484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:36.963727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:36.963690    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:37.610599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:37.610415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:38.342975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:38.342888    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:38.935652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:38.935618    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:39.505627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:39.482124    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:40.168190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:40.168121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:40.768598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:40.761597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:41.506612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:41.505368    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:42.304973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:42.304936    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:42.848053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:42.847963    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:43.093927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:43.093889    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:43.670031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:43.669983    5817 generic.go:356] "Generic (PLEG): container finished" podID="6f823003-51a6-438f-8270-5fe47685aa9e" containerID="59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92" exitCode=0
Nov 03 11:17:43.670781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:43.670752    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" event={"ID":"6f823003-51a6-438f-8270-5fe47685aa9e","Type":"ContainerDied","Data":"59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92"}
Nov 03 11:17:43.671553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:43.671530    5817 scope.go:122] "RemoveContainer" containerID="59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92"
Nov 03 11:17:43.675123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:43.675091    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:43.675287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:43.675272    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:43.675395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:43.675383    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:44.062862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:44.062803    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:44.686413 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:44.686022    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" event={"ID":"6f823003-51a6-438f-8270-5fe47685aa9e","Type":"ContainerStarted","Data":"fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6"}
Nov 03 11:17:44.754084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:44.754046    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:45.351426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:45.351394    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:17:45.353313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:45.353284    5817 patch_prober.go:29] interesting pod/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Nov 03 11:17:45.353472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:45.353343    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Nov 03 11:17:45.418598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:45.417513    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:46.068730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:46.068691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:46.353075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:46.352980    5817 patch_prober.go:29] interesting pod/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Nov 03 11:17:46.353075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:46.353032    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Nov 03 11:17:46.657452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:46.657260    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:47.371914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:47.369943    5817 patch_prober.go:29] interesting pod/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Nov 03 11:17:47.371914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:47.370023    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Nov 03 11:17:47.373454 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:47.373425    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:48.000595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:47.997560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:48.351927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:48.351834    5817 patch_prober.go:29] interesting pod/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Nov 03 11:17:48.351927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:48.351905    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Nov 03 11:17:48.670300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:48.668225    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:48.949891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:48.949866    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:17:49.353012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:49.352931    5817 patch_prober.go:29] interesting pod/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Nov 03 11:17:49.353012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:49.352994    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Nov 03 11:17:49.449936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:49.449900    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:50.075460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:50.075200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:50.358379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:50.358305    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:17:50.359336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:50.359320    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:17:50.718354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:50.718322    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:51.522545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:51.522511    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:52.426976 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:17:52.426933    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:17:52.550600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:52.544337    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:53.303606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:53.302883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:53.654649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:53.654543    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:53.988538 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:53.988503    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:54.130199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:54.130161    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:54.551191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:54.551145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:54.721596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:54.718782    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:55.588801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:55.588749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:56.366815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:56.359119    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:57.031934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:57.031902    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:57.582102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:57.582061    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:58.303611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:58.287818    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:59.030593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:59.028639    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:17:59.573285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:17:59.573239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:00.349057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:00.348599    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:00.919603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:00.916268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:01.591317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:01.591236    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:02.241777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:02.237941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:02.868626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:02.868581    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:03.787593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:03.781967    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:04.368451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:04.368413    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:05.318465 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:05.318432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:06.022764 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:06.022726    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:07.041314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:07.032510    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:07.276809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:07.276768    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:07.428179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:07.428144    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:07.915671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:07.909459    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:08.217699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.217665    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:08.474215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.474112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:08.550598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.549135    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"]
Nov 03 11:18:08.560931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.560904    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:08.562165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.562134    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"]
Nov 03 11:18:08.641819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.641783    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9143e1\""
Nov 03 11:18:09.129692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.653621    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97ng\" (UniqueName: \"kubernetes.io/projected/be9299d8-3824-4025-a353-119470c37edd-kube-api-access-j97ng\") pod \"payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9\" (UID: \"be9299d8-3824-4025-a353-119470c37edd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:09.140696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:09.130223    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9299d8-3824-4025-a353-119470c37edd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9\" (UID: \"be9299d8-3824-4025-a353-119470c37edd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:09.141925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.700917    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:09.142192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:08.654818    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9143e1-dockercfg-h8z6j\""
Nov 03 11:18:09.242729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:09.242700    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9299d8-3824-4025-a353-119470c37edd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9\" (UID: \"be9299d8-3824-4025-a353-119470c37edd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:09.243502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:09.243480    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9299d8-3824-4025-a353-119470c37edd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9\" (UID: \"be9299d8-3824-4025-a353-119470c37edd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:09.243707 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:09.243491    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97ng\" (UniqueName: \"kubernetes.io/projected/be9299d8-3824-4025-a353-119470c37edd-kube-api-access-j97ng\") pod \"payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9\" (UID: \"be9299d8-3824-4025-a353-119470c37edd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:09.252048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:09.252022    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97ng\" (UniqueName: \"kubernetes.io/projected/be9299d8-3824-4025-a353-119470c37edd-kube-api-access-j97ng\") pod \"payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9\" (UID: \"be9299d8-3824-4025-a353-119470c37edd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:09.483913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:09.483877    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:09.822361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:09.822213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:10.350653 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:18:10.350161    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9299d8_3824_4025_a353_119470c37edd.slice/crio-7935094a75fd188ceb6c1a296d9f7467e69b875817d51faebba2eba2cde96aa1 WatchSource:0}: Error finding container 7935094a75fd188ceb6c1a296d9f7467e69b875817d51faebba2eba2cde96aa1: Status 404 returned error can't find the container with id 7935094a75fd188ceb6c1a296d9f7467e69b875817d51faebba2eba2cde96aa1
Nov 03 11:18:10.354389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:10.354357    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"]
Nov 03 11:18:10.559602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:10.558230    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:11.280598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:11.277366    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:11.379960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:11.379928    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9" event={"ID":"be9299d8-3824-4025-a353-119470c37edd","Type":"ContainerStarted","Data":"e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65"}
Nov 03 11:18:11.380380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:11.379989    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9" event={"ID":"be9299d8-3824-4025-a353-119470c37edd","Type":"ContainerStarted","Data":"7935094a75fd188ceb6c1a296d9f7467e69b875817d51faebba2eba2cde96aa1"}
Nov 03 11:18:11.380380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:11.380067    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:11.398354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:11.398298    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9" podStartSLOduration=3.39828006 podStartE2EDuration="3.39828006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:18:11.395854205 +0000 UTC m=+2921.601436257" watchObservedRunningTime="2026-11-03 11:18:11.39828006 +0000 UTC m=+2921.603862112"
Nov 03 11:18:11.978938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:11.977379    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:12.199030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:12.198984    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:12.377812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:12.377711    5817 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:18:12.377812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:12.377773    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:18:12.618611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:12.617222    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:13.199582 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:13.199536    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:13.494719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:13.494683    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:13.945969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:13.945932    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:14.147965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:14.147932    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:14.369406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:14.369309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:15.085224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:15.085183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:15.850126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:15.850077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:16.455600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:16.452127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:17.140597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:17.134321    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:17.557490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:17.557450    5817 generic.go:356] "Generic (PLEG): container finished" podID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerID="98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05" exitCode=0
Nov 03 11:18:17.557943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:17.557531    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" event={"ID":"99eda1c3-5954-45d0-b159-d1f95545bd75","Type":"ContainerDied","Data":"98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05"}
Nov 03 11:18:17.558062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:17.558041    5817 scope.go:122] "RemoveContainer" containerID="98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05"
Nov 03 11:18:17.560166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:17.560136    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:18:17.560300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:17.560207    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:18:17.560300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:17.560241    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:18:17.894652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:17.884215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:18.515900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:18.515550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:18.652599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:18.649279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:18.691697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:18.691657    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" event={"ID":"99eda1c3-5954-45d0-b159-d1f95545bd75","Type":"ContainerStarted","Data":"5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67"}
Nov 03 11:18:19.231052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:19.230999    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:18:19.231304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:19.231280    5817 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 11:18:19.231374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:19.231351    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 11:18:19.268388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:19.268354    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:19.959597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:19.957223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:20.230650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:20.230541    5817 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 11:18:20.230650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:20.230605    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 11:18:20.691997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:20.691946    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:21.240017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:21.239982    5817 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 11:18:21.240726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:21.240688    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 11:18:21.441519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:21.441467    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:22.173624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:22.170032    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:22.243595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:22.241085    5817 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 11:18:22.243595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:22.241168    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 11:18:22.426261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:22.426180    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:18:22.454699 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:18:22.454666    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:18:22.891981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:22.891942    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:23.230792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:23.230757    5817 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 11:18:23.231109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:23.231081    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 11:18:23.643663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:23.643560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:24.247762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:24.247722    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:18:24.253687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:24.248918    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:18:24.253687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:24.250517    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:18:24.253687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:24.251072    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:24.387427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:24.387391    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:25.182598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:25.171338    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:25.953537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:25.953499    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:26.652606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:26.649678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:27.355595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:27.349930    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:28.097610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:28.097248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:28.916597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:28.913558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:29.762643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:29.761500    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:30.392697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:30.391642    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:31.065788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:31.055042    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:31.674602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:31.671003    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:32.283100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:32.283063    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:32.946618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:32.946564    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:33.094106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.094028    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:33.226656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.226527    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:33.650911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.650836    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"]
Nov 03 11:18:33.656166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.656142    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.659459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.659437    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:18:33.662680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.660586    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:18:33.662680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.662292    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"]
Nov 03 11:18:33.725279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.725248    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.725652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.725631    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2x5h\" (UniqueName: \"kubernetes.io/projected/65c59a54-c4d5-4e02-a940-ad458f72f559-kube-api-access-d2x5h\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.725896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.725881    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.827505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.827463    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.827725 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.827620    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2x5h\" (UniqueName: \"kubernetes.io/projected/65c59a54-c4d5-4e02-a940-ad458f72f559-kube-api-access-d2x5h\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.827796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.827740    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.828448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.828418    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.830603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.830545    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.836442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.836422    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2x5h\" (UniqueName: \"kubernetes.io/projected/65c59a54-c4d5-4e02-a940-ad458f72f559-kube-api-access-d2x5h\") pod \"maas-api-revoke-sub-eb846996-m42sb\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:33.889631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.889596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:33.982507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:33.982481    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:34.275011 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:18:34.274985    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c59a54_c4d5_4e02_a940_ad458f72f559.slice/crio-c1d857ec7d48577868821b459fc681ab06dab3ab912e7e7e1274c533645e50d2 WatchSource:0}: Error finding container c1d857ec7d48577868821b459fc681ab06dab3ab912e7e7e1274c533645e50d2: Status 404 returned error can't find the container with id c1d857ec7d48577868821b459fc681ab06dab3ab912e7e7e1274c533645e50d2
Nov 03 11:18:34.278756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:34.278729    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"]
Nov 03 11:18:34.563651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:34.557324    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:34.843304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:34.843240    5817 scope.go:122] "RemoveContainer" containerID="2dbd965b4b04c2fd2493eda6abc08aed61edad6fe1ddcc4c5b89b7d1a0aaab5a"
Nov 03 11:18:35.161299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:35.161204    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb" event={"ID":"65c59a54-c4d5-4e02-a940-ad458f72f559","Type":"ContainerStarted","Data":"3b8d22e936e90b0c0e8e3151c13894d4bed0a64f73832613d11be190502e5bf6"}
Nov 03 11:18:35.161527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:35.161316    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb" event={"ID":"65c59a54-c4d5-4e02-a940-ad458f72f559","Type":"ContainerStarted","Data":"c1d857ec7d48577868821b459fc681ab06dab3ab912e7e7e1274c533645e50d2"}
Nov 03 11:18:35.178922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:35.178867    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb" podStartSLOduration=2.178852089 podStartE2EDuration="2.178852089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:18:35.177092208 +0000 UTC m=+2945.382674261" watchObservedRunningTime="2026-11-03 11:18:35.178852089 +0000 UTC m=+2945.384434142"
Nov 03 11:18:35.246212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:35.246173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:35.851604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:35.845098    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:36.172957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:36.172882    5817 generic.go:356] "Generic (PLEG): container finished" podID="65c59a54-c4d5-4e02-a940-ad458f72f559" containerID="3b8d22e936e90b0c0e8e3151c13894d4bed0a64f73832613d11be190502e5bf6" exitCode=0
Nov 03 11:18:36.173113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:36.172967    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb" event={"ID":"65c59a54-c4d5-4e02-a940-ad458f72f559","Type":"ContainerDied","Data":"3b8d22e936e90b0c0e8e3151c13894d4bed0a64f73832613d11be190502e5bf6"}
Nov 03 11:18:36.649206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:36.649169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:37.370791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.369744    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:37.463141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.463070    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:37.476016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.475979    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-openshift-ca\") pod \"65c59a54-c4d5-4e02-a940-ad458f72f559\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") "
Nov 03 11:18:37.476213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.476194    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-xks-ca\") pod \"65c59a54-c4d5-4e02-a940-ad458f72f559\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") "
Nov 03 11:18:37.476409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.476383    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c59a54-c4d5-4e02-a940-ad458f72f559-kube-api-access-d2x5h\" (UniqueName: \"kubernetes.io/projected/65c59a54-c4d5-4e02-a940-ad458f72f559-kube-api-access-d2x5h\") pod \"65c59a54-c4d5-4e02-a940-ad458f72f559\" (UID: \"65c59a54-c4d5-4e02-a940-ad458f72f559\") "
Nov 03 11:18:37.476702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.476670    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-openshift-ca" pod "65c59a54-c4d5-4e02-a940-ad458f72f559" (UID: "65c59a54-c4d5-4e02-a940-ad458f72f559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:18:37.477283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.477208    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:37.479443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.479414    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-xks-ca" pod "65c59a54-c4d5-4e02-a940-ad458f72f559" (UID: "65c59a54-c4d5-4e02-a940-ad458f72f559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:18:37.480830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.480773    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c59a54-c4d5-4e02-a940-ad458f72f559-kube-api-access-d2x5h" pod "65c59a54-c4d5-4e02-a940-ad458f72f559" (UID: "65c59a54-c4d5-4e02-a940-ad458f72f559"). InnerVolumeSpecName "kube-api-access-d2x5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:18:37.579617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.579586    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c59a54-c4d5-4e02-a940-ad458f72f559-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:37.579617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:37.579614    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2x5h\" (UniqueName: \"kubernetes.io/projected/65c59a54-c4d5-4e02-a940-ad458f72f559-kube-api-access-d2x5h\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:38.049044 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:38.049008    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:38.194530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:38.194502    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb" event={"ID":"65c59a54-c4d5-4e02-a940-ad458f72f559","Type":"ContainerDied","Data":"c1d857ec7d48577868821b459fc681ab06dab3ab912e7e7e1274c533645e50d2"}
Nov 03 11:18:38.194758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:38.194535    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"
Nov 03 11:18:38.194758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:38.194549    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d857ec7d48577868821b459fc681ab06dab3ab912e7e7e1274c533645e50d2"
Nov 03 11:18:38.669640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:38.669531    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:40.325944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:40.325882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:40.676033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:40.675930    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:41.485620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:41.485583    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:42.088547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:42.079513    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:42.708603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:42.704414    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:43.375385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:43.375288    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:44.068604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.066201    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:44.450974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.450878    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:44.852925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.852854    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"]
Nov 03 11:18:44.853957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.853926    5817 cpu_manager.go:414] "containerMap: removing container" podUID="65c59a54-c4d5-4e02-a940-ad458f72f559" containerName="revoke-keys"
Nov 03 11:18:44.853957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.853946    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c59a54-c4d5-4e02-a940-ad458f72f559" containerName="revoke-keys"
Nov 03 11:18:44.854165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.854089    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c59a54-c4d5-4e02-a940-ad458f72f559" containerName="revoke-keys"
Nov 03 11:18:44.858539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.858514    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:44.861975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.861940    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:18:44.863194 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.863167    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:18:44.865006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.864983    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"]
Nov 03 11:18:44.969586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.969550    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:44.969786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.969648    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:44.969867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:44.969846    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68g29\" (UniqueName: \"kubernetes.io/projected/9c96d741-f606-49f3-8b41-3e85be4f1558-kube-api-access-68g29\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:45.037513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.037479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:45.071872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.071832    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:45.085722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.085685    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:45.086144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.086117    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68g29\" (UniqueName: \"kubernetes.io/projected/9c96d741-f606-49f3-8b41-3e85be4f1558-kube-api-access-68g29\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:45.102826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.102788    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:45.127025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.125028    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:45.129191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.129128    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68g29\" (UniqueName: \"kubernetes.io/projected/9c96d741-f606-49f3-8b41-3e85be4f1558-kube-api-access-68g29\") pod \"maas-api-revoke-sub-857df2ab-l5xxz\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:45.183336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.183306    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:45.481134 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:18:45.480856    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c96d741_f606_49f3_8b41_3e85be4f1558.slice/crio-1ecdd612c32e99eb2433a7dc6bf3f3307e7d782263ec8e0f6e38f626a0ac8a73 WatchSource:0}: Error finding container 1ecdd612c32e99eb2433a7dc6bf3f3307e7d782263ec8e0f6e38f626a0ac8a73: Status 404 returned error can't find the container with id 1ecdd612c32e99eb2433a7dc6bf3f3307e7d782263ec8e0f6e38f626a0ac8a73
Nov 03 11:18:45.481134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.481046    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"]
Nov 03 11:18:45.841706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:45.841173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:46.301121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:46.301069    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz" event={"ID":"9c96d741-f606-49f3-8b41-3e85be4f1558","Type":"ContainerStarted","Data":"247eb96267f98400f83569f71636ce725b2e789dec6e6d218e5e2fd3587741f3"}
Nov 03 11:18:46.301587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:46.301150    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz" event={"ID":"9c96d741-f606-49f3-8b41-3e85be4f1558","Type":"ContainerStarted","Data":"1ecdd612c32e99eb2433a7dc6bf3f3307e7d782263ec8e0f6e38f626a0ac8a73"}
Nov 03 11:18:46.324320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:46.324268    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz" podStartSLOduration=2.324251062 podStartE2EDuration="2.324251062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:18:46.315767488 +0000 UTC m=+2956.521349540" watchObservedRunningTime="2026-11-03 11:18:46.324251062 +0000 UTC m=+2956.529833114"
Nov 03 11:18:46.654384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:46.654238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:47.314891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:47.314854    5817 generic.go:356] "Generic (PLEG): container finished" podID="9c96d741-f606-49f3-8b41-3e85be4f1558" containerID="247eb96267f98400f83569f71636ce725b2e789dec6e6d218e5e2fd3587741f3" exitCode=0
Nov 03 11:18:47.315406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:47.314950    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz" event={"ID":"9c96d741-f606-49f3-8b41-3e85be4f1558","Type":"ContainerDied","Data":"247eb96267f98400f83569f71636ce725b2e789dec6e6d218e5e2fd3587741f3"}
Nov 03 11:18:47.414803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:47.414764    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:48.023159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.023119    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"]
Nov 03 11:18:48.029086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.029061    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea9d6ee-l264b"]
Nov 03 11:18:48.087866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.087832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:48.517254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.517229    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:48.600764 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.600668    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c96d741-f606-49f3-8b41-3e85be4f1558-kube-api-access-68g29\" (UniqueName: \"kubernetes.io/projected/9c96d741-f606-49f3-8b41-3e85be4f1558-kube-api-access-68g29\") pod \"9c96d741-f606-49f3-8b41-3e85be4f1558\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") "
Nov 03 11:18:48.600927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.600872    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-xks-ca\") pod \"9c96d741-f606-49f3-8b41-3e85be4f1558\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") "
Nov 03 11:18:48.601198 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.601173    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-openshift-ca\") pod \"9c96d741-f606-49f3-8b41-3e85be4f1558\" (UID: \"9c96d741-f606-49f3-8b41-3e85be4f1558\") "
Nov 03 11:18:48.601872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.601837    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-openshift-ca" pod "9c96d741-f606-49f3-8b41-3e85be4f1558" (UID: "9c96d741-f606-49f3-8b41-3e85be4f1558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:18:48.602263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.602237    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:48.604029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.603994    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c96d741-f606-49f3-8b41-3e85be4f1558-kube-api-access-68g29" pod "9c96d741-f606-49f3-8b41-3e85be4f1558" (UID: "9c96d741-f606-49f3-8b41-3e85be4f1558"). InnerVolumeSpecName "kube-api-access-68g29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:18:48.604657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.604629    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-xks-ca" pod "9c96d741-f606-49f3-8b41-3e85be4f1558" (UID: "9c96d741-f606-49f3-8b41-3e85be4f1558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:18:48.703477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.703450    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68g29\" (UniqueName: \"kubernetes.io/projected/9c96d741-f606-49f3-8b41-3e85be4f1558-kube-api-access-68g29\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:48.703477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.703477    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96d741-f606-49f3-8b41-3e85be4f1558-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:48.927654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:48.926686    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:49.198353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:49.198320    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:18:49.479649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:49.476308    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz" event={"ID":"9c96d741-f606-49f3-8b41-3e85be4f1558","Type":"ContainerDied","Data":"1ecdd612c32e99eb2433a7dc6bf3f3307e7d782263ec8e0f6e38f626a0ac8a73"}
Nov 03 11:18:49.479649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:49.476372    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ecdd612c32e99eb2433a7dc6bf3f3307e7d782263ec8e0f6e38f626a0ac8a73"
Nov 03 11:18:49.479649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:49.476445    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"
Nov 03 11:18:49.899645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:49.891204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:49.956622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:49.956536    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05f4553-02e1-464a-8690-eee4089fad50" path="/var/lib/kubelet/pods/f05f4553-02e1-464a-8690-eee4089fad50/volumes"
Nov 03 11:18:50.767414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:50.767381    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:51.321602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:51.312229    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:51.454608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:51.452111    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:52.246595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:52.246089    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:52.437493 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:18:52.437453    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:18:52.961606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:52.959589    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:53.597967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:53.591835    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:54.125077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:54.125044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:54.369596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:54.366111    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:55.245189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:55.245156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:55.881410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:55.881377    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:56.544605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:56.543092    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:57.137648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:57.137555    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:57.868596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:57.866172    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:58.457600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:58.456706    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:59.026640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:59.026588    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"]
Nov 03 11:18:59.032249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:59.032214    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d197c3-8st2f"]
Nov 03 11:18:59.104611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:59.099833    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:59.866394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:59.866359    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:18:59.934623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:18:59.934591    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0f2b24-bed5-4978-aa5d-a21bfd6b7117" path="/var/lib/kubelet/pods/1b0f2b24-bed5-4978-aa5d-a21bfd6b7117/volumes"
Nov 03 11:19:00.462601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:00.458295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:01.174353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:01.174314    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:01.854405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:01.854362    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:02.477782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:02.477737    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:03.261611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:03.256361    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:03.853720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:03.847063    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:04.580260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:04.580157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:05.225556 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.225517    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:05.643590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.643483    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:05.855808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.855780    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"]
Nov 03 11:19:05.856857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.856838    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9c96d741-f606-49f3-8b41-3e85be4f1558" containerName="revoke-keys"
Nov 03 11:19:05.856857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.856857    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c96d741-f606-49f3-8b41-3e85be4f1558" containerName="revoke-keys"
Nov 03 11:19:05.857015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.857005    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c96d741-f606-49f3-8b41-3e85be4f1558" containerName="revoke-keys"
Nov 03 11:19:05.861549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.861527    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:05.864759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.864720    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:19:05.865921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.865901    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:19:05.867732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.867708    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"]
Nov 03 11:19:05.932438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.932411    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:05.932639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.932600    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgk8h\" (UniqueName: \"kubernetes.io/projected/5c3afe20-a233-4dbb-8176-d40f1a7261df-kube-api-access-xgk8h\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:05.932720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:05.932695    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:06.034130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.034094    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:06.034332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.034263    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgk8h\" (UniqueName: \"kubernetes.io/projected/5c3afe20-a233-4dbb-8176-d40f1a7261df-kube-api-access-xgk8h\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:06.034421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.034345    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:06.035032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.035007    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:06.037095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.037069    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:06.053164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.053140    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgk8h\" (UniqueName: \"kubernetes.io/projected/5c3afe20-a233-4dbb-8176-d40f1a7261df-kube-api-access-xgk8h\") pod \"maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:06.183170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.183105    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:06.309596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.306182    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:06.472445 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:19:06.472410    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3afe20_a233_4dbb_8176_d40f1a7261df.slice/crio-b81410b0bcd2fe1648a227bd0cca5e6a82fe29a06138ccbc55ae1c6856f2328c WatchSource:0}: Error finding container b81410b0bcd2fe1648a227bd0cca5e6a82fe29a06138ccbc55ae1c6856f2328c: Status 404 returned error can't find the container with id b81410b0bcd2fe1648a227bd0cca5e6a82fe29a06138ccbc55ae1c6856f2328c
Nov 03 11:19:06.474034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.474009    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"]
Nov 03 11:19:06.824866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:06.824752    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn" event={"ID":"5c3afe20-a233-4dbb-8176-d40f1a7261df","Type":"ContainerStarted","Data":"b81410b0bcd2fe1648a227bd0cca5e6a82fe29a06138ccbc55ae1c6856f2328c"}
Nov 03 11:19:07.259255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:07.259211    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:07.838626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:07.838533    5817 generic.go:356] "Generic (PLEG): container finished" podID="5c3afe20-a233-4dbb-8176-d40f1a7261df" containerID="e8d48fe03add1353294ce73c48d1466c71e3ec950b953f36dba9d61d9588cc15" exitCode=0
Nov 03 11:19:07.839112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:07.838726    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn" event={"ID":"5c3afe20-a233-4dbb-8176-d40f1a7261df","Type":"ContainerDied","Data":"e8d48fe03add1353294ce73c48d1466c71e3ec950b953f36dba9d61d9588cc15"}
Nov 03 11:19:08.072597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:08.067707    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:08.255578 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:08.255541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:08.778600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:08.772210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:09.062069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.062040    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:09.145075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.145041    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3afe20-a233-4dbb-8176-d40f1a7261df-kube-api-access-xgk8h\" (UniqueName: \"kubernetes.io/projected/5c3afe20-a233-4dbb-8176-d40f1a7261df-kube-api-access-xgk8h\") pod \"5c3afe20-a233-4dbb-8176-d40f1a7261df\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") "
Nov 03 11:19:09.145285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.145091    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-openshift-ca\") pod \"5c3afe20-a233-4dbb-8176-d40f1a7261df\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") "
Nov 03 11:19:09.145527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.145495    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-openshift-ca" pod "5c3afe20-a233-4dbb-8176-d40f1a7261df" (UID: "5c3afe20-a233-4dbb-8176-d40f1a7261df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:19:09.145667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.145564    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-xks-ca\") pod \"5c3afe20-a233-4dbb-8176-d40f1a7261df\" (UID: \"5c3afe20-a233-4dbb-8176-d40f1a7261df\") "
Nov 03 11:19:09.146296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.146274    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:09.148445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.148412    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-xks-ca" pod "5c3afe20-a233-4dbb-8176-d40f1a7261df" (UID: "5c3afe20-a233-4dbb-8176-d40f1a7261df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:19:09.148619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.148523    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3afe20-a233-4dbb-8176-d40f1a7261df-kube-api-access-xgk8h" pod "5c3afe20-a233-4dbb-8176-d40f1a7261df" (UID: "5c3afe20-a233-4dbb-8176-d40f1a7261df"). InnerVolumeSpecName "kube-api-access-xgk8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:09.248646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.248613    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3afe20-a233-4dbb-8176-d40f1a7261df-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:09.248646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.248645    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgk8h\" (UniqueName: \"kubernetes.io/projected/5c3afe20-a233-4dbb-8176-d40f1a7261df-kube-api-access-xgk8h\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:09.497093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.497050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:09.873742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.873662    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"
Nov 03 11:19:09.873742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.873697    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn" event={"ID":"5c3afe20-a233-4dbb-8176-d40f1a7261df","Type":"ContainerDied","Data":"b81410b0bcd2fe1648a227bd0cca5e6a82fe29a06138ccbc55ae1c6856f2328c"}
Nov 03 11:19:09.873937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:09.873746    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81410b0bcd2fe1648a227bd0cca5e6a82fe29a06138ccbc55ae1c6856f2328c"
Nov 03 11:19:10.185677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:10.182105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:10.887549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:10.887515    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:11.030407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:11.030374    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"]
Nov 03 11:19:11.038139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:11.038115    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93a8ab-nkmr9"]
Nov 03 11:19:11.774733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:11.768558    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:11.907940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:11.907903    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cfd3202-6582-471c-be21-211b5ace4aee" path="/var/lib/kubelet/pods/2cfd3202-6582-471c-be21-211b5ace4aee/volumes"
Nov 03 11:19:12.490691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:12.488477    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:13.231244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:13.231206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:13.944318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:13.942212    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:14.584283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:14.584237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:15.240027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:15.239990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:15.871909 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:15.871871    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:16.135740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:16.129183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:16.297600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:16.292272    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:16.997602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:16.996694    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:17.846667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:17.846623    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:18.461831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:18.461798    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"]
Nov 03 11:19:18.462856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:18.462827    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7" podUID="7c2abdf7-0efa-47ef-ba7c-70819dfbf167" containerName="maas-api" containerID="cri-o://934fc27f52fc4d1a2c514f8e2917f406afe1b623b9a9196528834d3fbac6bc14" gracePeriod=30
Nov 03 11:19:18.550212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:18.550177    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:18.596794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:18.596752    5817 generic.go:356] "Generic (PLEG): container finished" podID="7c2abdf7-0efa-47ef-ba7c-70819dfbf167" containerID="934fc27f52fc4d1a2c514f8e2917f406afe1b623b9a9196528834d3fbac6bc14" exitCode=0
Nov 03 11:19:18.596910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:18.596832    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7" event={"ID":"7c2abdf7-0efa-47ef-ba7c-70819dfbf167","Type":"ContainerDied","Data":"934fc27f52fc4d1a2c514f8e2917f406afe1b623b9a9196528834d3fbac6bc14"}
Nov 03 11:19:18.710596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:18.709047    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:18.954322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:18.913128    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:19.205714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.205684    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"]
Nov 03 11:19:19.213680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.213266    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9265bf-n6jjn"]
Nov 03 11:19:19.379500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.379460    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:19.393837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.393812    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:19:19.462779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.462700    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"]
Nov 03 11:19:19.464308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.464285    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5c3afe20-a233-4dbb-8176-d40f1a7261df" containerName="revoke-keys"
Nov 03 11:19:19.464386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.464312    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3afe20-a233-4dbb-8176-d40f1a7261df" containerName="revoke-keys"
Nov 03 11:19:19.464386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.464334    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7c2abdf7-0efa-47ef-ba7c-70819dfbf167" containerName="maas-api"
Nov 03 11:19:19.464386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.464342    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2abdf7-0efa-47ef-ba7c-70819dfbf167" containerName="maas-api"
Nov 03 11:19:19.464613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.464595    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3afe20-a233-4dbb-8176-d40f1a7261df" containerName="revoke-keys"
Nov 03 11:19:19.464661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.464615    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2abdf7-0efa-47ef-ba7c-70819dfbf167" containerName="maas-api"
Nov 03 11:19:19.469324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.469307    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.479668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.479648    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9143e1-serving-cert\""
Nov 03 11:19:19.490309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.490286    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"]
Nov 03 11:19:19.505463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.505434    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-kube-api-access-gkx8j\" (UniqueName: \"kubernetes.io/projected/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-kube-api-access-gkx8j\") pod \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") "
Nov 03 11:19:19.505661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.505544    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-maas-api-tls\") pod \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") "
Nov 03 11:19:19.505661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.505613    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-openshift-service-ca\") pod \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") "
Nov 03 11:19:19.505837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.505818    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-metrics-certs\") pod \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\" (UID: \"7c2abdf7-0efa-47ef-ba7c-70819dfbf167\") "
Nov 03 11:19:19.505991 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.505965    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-openshift-service-ca" pod "7c2abdf7-0efa-47ef-ba7c-70819dfbf167" (UID: "7c2abdf7-0efa-47ef-ba7c-70819dfbf167"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:19:19.506617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.506550    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5bd\" (UniqueName: \"kubernetes.io/projected/2620222d-9f11-40c4-b27d-e15d59be1755-kube-api-access-5g5bd\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.506799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.506774    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-metrics-certs\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.506941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.506870    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-maas-api-tls\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.506941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.506924    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2620222d-9f11-40c4-b27d-e15d59be1755-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.507414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.507317    5817 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:19.508870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.508837    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-metrics-certs" pod "7c2abdf7-0efa-47ef-ba7c-70819dfbf167" (UID: "7c2abdf7-0efa-47ef-ba7c-70819dfbf167"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:19:19.509025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.509008    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-maas-api-tls" pod "7c2abdf7-0efa-47ef-ba7c-70819dfbf167" (UID: "7c2abdf7-0efa-47ef-ba7c-70819dfbf167"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:19:19.509199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.509175    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-kube-api-access-gkx8j" pod "7c2abdf7-0efa-47ef-ba7c-70819dfbf167" (UID: "7c2abdf7-0efa-47ef-ba7c-70819dfbf167"). InnerVolumeSpecName "kube-api-access-gkx8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:19.608835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.608803    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5bd\" (UniqueName: \"kubernetes.io/projected/2620222d-9f11-40c4-b27d-e15d59be1755-kube-api-access-5g5bd\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.609050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.608943    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-metrics-certs\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.609050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.608985    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"
Nov 03 11:19:19.609050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.609019    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-maas-api-tls\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.609217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.609068    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2620222d-9f11-40c4-b27d-e15d59be1755-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.609217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.609194    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7" event={"ID":"7c2abdf7-0efa-47ef-ba7c-70819dfbf167","Type":"ContainerDied","Data":"2d4bdb33854850ba8a314e5a4334b8aae4d20c08480e1dc86307e7cc8bf2454b"}
Nov 03 11:19:19.609337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.609262    5817 scope.go:122] "RemoveContainer" containerID="934fc27f52fc4d1a2c514f8e2917f406afe1b623b9a9196528834d3fbac6bc14"
Nov 03 11:19:19.609540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.609514    5817 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:19.609735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.609654    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkx8j\" (UniqueName: \"kubernetes.io/projected/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-kube-api-access-gkx8j\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:19.609735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.609679    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2abdf7-0efa-47ef-ba7c-70819dfbf167-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:19.610198 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.610179    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2620222d-9f11-40c4-b27d-e15d59be1755-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.613065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.613046    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-metrics-certs\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.613332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.613312    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-maas-api-tls\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.617525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.617507    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5bd\" (UniqueName: \"kubernetes.io/projected/2620222d-9f11-40c4-b27d-e15d59be1755-kube-api-access-5g5bd\") pod \"maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.670821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.670800    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"]
Nov 03 11:19:19.676386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.676367    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9265bf-5c847588f8-qmbx7"]
Nov 03 11:19:19.680925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.680900    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:19.788662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.788563    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:19.913056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.913025    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3afe20-a233-4dbb-8176-d40f1a7261df" path="/var/lib/kubelet/pods/5c3afe20-a233-4dbb-8176-d40f1a7261df/volumes"
Nov 03 11:19:19.913840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:19.913820    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2abdf7-0efa-47ef-ba7c-70819dfbf167" path="/var/lib/kubelet/pods/7c2abdf7-0efa-47ef-ba7c-70819dfbf167/volumes"
Nov 03 11:19:20.101623 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:19:20.101547    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2620222d_9f11_40c4_b27d_e15d59be1755.slice/crio-9b30cec62f95f0da6b4f3b72a446d5730307af803ab89112d80f782f07d672f9 WatchSource:0}: Error finding container 9b30cec62f95f0da6b4f3b72a446d5730307af803ab89112d80f782f07d672f9: Status 404 returned error can't find the container with id 9b30cec62f95f0da6b4f3b72a446d5730307af803ab89112d80f782f07d672f9
Nov 03 11:19:20.103200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:20.103179    5817 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:19:20.106692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:20.106667    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"]
Nov 03 11:19:20.289660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:20.280699    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:20.623988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:20.623950    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5" event={"ID":"2620222d-9f11-40c4-b27d-e15d59be1755","Type":"ContainerStarted","Data":"9b30cec62f95f0da6b4f3b72a446d5730307af803ab89112d80f782f07d672f9"}
Nov 03 11:19:21.016598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:21.015221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:21.641340 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:21.641305    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5" event={"ID":"2620222d-9f11-40c4-b27d-e15d59be1755","Type":"ContainerStarted","Data":"682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a"}
Nov 03 11:19:21.641840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:21.641381    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:21.664226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:21.664179    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5" podStartSLOduration=49.155158854 podStartE2EDuration="49.664166297s" totalImagesPullingTime="509.007443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:19:21.657800623 +0000 UTC m=+2991.863382674" watchObservedRunningTime="2026-11-03 11:19:21.664166297 +0000 UTC m=+2991.869748365"
Nov 03 11:19:21.829906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:21.828401    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:22.415309 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:19:22.415271    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:19:22.474649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:22.460409    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:23.027024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:23.026975    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"]
Nov 03 11:19:23.032688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:23.032659    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03aa334b-jfg67"]
Nov 03 11:19:23.193600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:23.193139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:23.905314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:23.905280    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724ad1c9-86b8-432c-a4d6-f61ea421a45d" path="/var/lib/kubelet/pods/724ad1c9-86b8-432c-a4d6-f61ea421a45d/volumes"
Nov 03 11:19:23.941289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:23.941184    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:24.580259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:24.580226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:25.359867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:25.359796    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:25.594397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:25.594356    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:25.745465 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:25.745431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:25.877687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:25.877654    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"]
Nov 03 11:19:25.877946 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:25.877921    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy" containerID="cri-o://908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f" gracePeriod=30
Nov 03 11:19:25.930444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:25.930409    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:26.069692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.067347    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:26.406668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.402936    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"]
Nov 03 11:19:26.410281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.410144    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.432484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.432456    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:19:26.432757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.432738    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:19:26.432856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.432834    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hz4d\" (UniqueName: \"kubernetes.io/projected/e73669a9-a186-485c-a0a9-09d68f9fb3a9-kube-api-access-4hz4d\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.433174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.433151    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.433251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.433227    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.433644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.433626    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"]
Nov 03 11:19:26.535144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.535101    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.535337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.535213    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.535337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.535249    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hz4d\" (UniqueName: \"kubernetes.io/projected/e73669a9-a186-485c-a0a9-09d68f9fb3a9-kube-api-access-4hz4d\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.535899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.535863    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.538145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.538116    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.553817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.553787    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hz4d\" (UniqueName: \"kubernetes.io/projected/e73669a9-a186-485c-a0a9-09d68f9fb3a9-kube-api-access-4hz4d\") pod \"maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:26.742562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:26.742528    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:27.087879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:27.085907    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:27.157257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:27.131412    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"]
Nov 03 11:19:27.657226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:27.657154    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:19:27.722267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:27.721765    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v" event={"ID":"e73669a9-a186-485c-a0a9-09d68f9fb3a9","Type":"ContainerStarted","Data":"4d253c3e9d7e772e063f1b7dafeccd654d0aa3f0cb0008dc54475d01675b7b3e"}
Nov 03 11:19:27.722267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:27.721834    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v" event={"ID":"e73669a9-a186-485c-a0a9-09d68f9fb3a9","Type":"ContainerStarted","Data":"6757fd0e804e86d2c1cc0a13b2818e4f9825b4814b6656efcf735c14e053b6d6"}
Nov 03 11:19:27.741835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:27.741119    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v" podStartSLOduration=1.7411035990000001 podStartE2EDuration="1.741103599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:19:27.740615516 +0000 UTC m=+2997.946197568" watchObservedRunningTime="2026-11-03 11:19:27.741103599 +0000 UTC m=+2997.946685650"
Nov 03 11:19:27.760062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:27.754986    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:28.523310 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:28.523186    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:28.735348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:28.735289    5817 generic.go:356] "Generic (PLEG): container finished" podID="e73669a9-a186-485c-a0a9-09d68f9fb3a9" containerID="4d253c3e9d7e772e063f1b7dafeccd654d0aa3f0cb0008dc54475d01675b7b3e" exitCode=0
Nov 03 11:19:28.735551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:28.735473    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v" event={"ID":"e73669a9-a186-485c-a0a9-09d68f9fb3a9","Type":"ContainerDied","Data":"4d253c3e9d7e772e063f1b7dafeccd654d0aa3f0cb0008dc54475d01675b7b3e"}
Nov 03 11:19:29.274594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:29.271312    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:29.969230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:29.967231    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:30.046675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.046648    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:30.073018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.072982    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-xks-ca\") pod \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") "
Nov 03 11:19:30.073396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.073365    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73669a9-a186-485c-a0a9-09d68f9fb3a9-kube-api-access-4hz4d\" (UniqueName: \"kubernetes.io/projected/e73669a9-a186-485c-a0a9-09d68f9fb3a9-kube-api-access-4hz4d\") pod \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") "
Nov 03 11:19:30.073504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.073422    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-openshift-ca\") pod \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\" (UID: \"e73669a9-a186-485c-a0a9-09d68f9fb3a9\") "
Nov 03 11:19:30.074423 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.074399    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-openshift-ca" pod "e73669a9-a186-485c-a0a9-09d68f9fb3a9" (UID: "e73669a9-a186-485c-a0a9-09d68f9fb3a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:19:30.078657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.075995    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:30.078657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.076319    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73669a9-a186-485c-a0a9-09d68f9fb3a9-kube-api-access-4hz4d" pod "e73669a9-a186-485c-a0a9-09d68f9fb3a9" (UID: "e73669a9-a186-485c-a0a9-09d68f9fb3a9"). InnerVolumeSpecName "kube-api-access-4hz4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:30.078657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.076847    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-xks-ca" pod "e73669a9-a186-485c-a0a9-09d68f9fb3a9" (UID: "e73669a9-a186-485c-a0a9-09d68f9fb3a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:19:30.177482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.177445    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hz4d\" (UniqueName: \"kubernetes.io/projected/e73669a9-a186-485c-a0a9-09d68f9fb3a9-kube-api-access-4hz4d\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:30.177482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.177477    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73669a9-a186-485c-a0a9-09d68f9fb3a9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:30.642599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.639560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:30.759619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.759541    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"
Nov 03 11:19:30.759823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.759542    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v" event={"ID":"e73669a9-a186-485c-a0a9-09d68f9fb3a9","Type":"ContainerDied","Data":"6757fd0e804e86d2c1cc0a13b2818e4f9825b4814b6656efcf735c14e053b6d6"}
Nov 03 11:19:30.759823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:30.759783    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6757fd0e804e86d2c1cc0a13b2818e4f9825b4814b6656efcf735c14e053b6d6"
Nov 03 11:19:31.206489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.206468    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:19:31.244241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.244213    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"]
Nov 03 11:19:31.245158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.245137    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e73669a9-a186-485c-a0a9-09d68f9fb3a9" containerName="revoke-keys"
Nov 03 11:19:31.245158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.245157    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73669a9-a186-485c-a0a9-09d68f9fb3a9" containerName="revoke-keys"
Nov 03 11:19:31.245353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.245188    5817 cpu_manager.go:414] "containerMap: removing container" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy"
Nov 03 11:19:31.245353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.245193    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy"
Nov 03 11:19:31.245441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.245382    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73669a9-a186-485c-a0a9-09d68f9fb3a9" containerName="revoke-keys"
Nov 03 11:19:31.245441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.245397    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy"
Nov 03 11:19:31.245812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.245791    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy"
Nov 03 11:19:31.260595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.258786    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.294449    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-kube-api-access-tfr8t\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-kube-api-access-tfr8t\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.294530    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-istio-token\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-istio-token\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.294830    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/406a01a7-6c91-4db3-a511-323ede863155-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406a01a7-6c91-4db3-a511-323ede863155-istio-podinfo\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.294896    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-envoy\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.295013    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-socket\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.295269    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-data\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.295322    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406a01a7-6c91-4db3-a511-323ede863155-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406a01a7-6c91-4db3-a511-323ede863155-istiod-ca-cert\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.295437    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-credential-socket\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.295469    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-certs\") pod \"406a01a7-6c91-4db3-a511-323ede863155\" (UID: \"406a01a7-6c91-4db3-a511-323ede863155\") "
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.295947    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296030    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-envoy\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296154    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-credential-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296266    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-podinfo\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296402    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-token\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296468    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnpwl\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-kube-api-access-bnpwl\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296659    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24376f-openshift-default-dockercfg-psf48\""
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296738    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-data\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296830    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-certs\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296898    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istiod-ca-cert\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.296963    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-credential-socket" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.313308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.309213    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-certs" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.314775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.313660    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-socket" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.341693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.315454    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406a01a7-6c91-4db3-a511-323ede863155-istiod-ca-cert" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:19:31.341693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.329693    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-data" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.341693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.336615    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"]
Nov 03 11:19:31.361775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.361740    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-envoy" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.381113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.381076    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400273    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-credential-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400369    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-podinfo\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400421    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-token\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400473    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnpwl\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-kube-api-access-bnpwl\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400671    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-data\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400742    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-certs\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400798    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istiod-ca-cert\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400936    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.400991    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-envoy\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.401114    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.401131    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.401148    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.401161    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.401175    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406a01a7-6c91-4db3-a511-323ede863155-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.401188    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406a01a7-6c91-4db3-a511-323ede863155-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.402441    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-istio-token" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.403148    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-credential-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.403662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.403638    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-certs\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.404921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.403900    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.404921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.404167    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-data\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.404921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.404791    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istiod-ca-cert\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.406549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.406522    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-kube-api-access-tfr8t" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "kube-api-access-tfr8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:31.406746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.406602    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-envoy\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.407198 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.407158    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/406a01a7-6c91-4db3-a511-323ede863155-istio-podinfo" pod "406a01a7-6c91-4db3-a511-323ede863155" (UID: "406a01a7-6c91-4db3-a511-323ede863155"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:19:31.411352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.410712    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-podinfo\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.415228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.415164    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnpwl\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-kube-api-access-bnpwl\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.415436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.415414    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-token\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.444614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.444589    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:31.502801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.502769    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfr8t\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-kube-api-access-tfr8t\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.502801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.502801    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406a01a7-6c91-4db3-a511-323ede863155-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.503060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.502817    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406a01a7-6c91-4db3-a511-323ede863155-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.747525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.747495    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"]
Nov 03 11:19:31.747870 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:19:31.747533    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa54aa54_6c59_4169_a0f2_beb2ca32557b.slice/crio-f40126d39a3e05cd1a3f20daa0753626a04ed46af3ebeb3cde23899db6e254e3 WatchSource:0}: Error finding container f40126d39a3e05cd1a3f20daa0753626a04ed46af3ebeb3cde23899db6e254e3: Status 404 returned error can't find the container with id f40126d39a3e05cd1a3f20daa0753626a04ed46af3ebeb3cde23899db6e254e3
Nov 03 11:19:31.750842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.750802    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:19:31.750979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.750871    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:19:31.750979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.750911    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:19:31.771382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.771356    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" event={"ID":"aa54aa54-6c59-4169-a0f2-beb2ca32557b","Type":"ContainerStarted","Data":"f40126d39a3e05cd1a3f20daa0753626a04ed46af3ebeb3cde23899db6e254e3"}
Nov 03 11:19:31.773310 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.773286    5817 generic.go:356] "Generic (PLEG): container finished" podID="406a01a7-6c91-4db3-a511-323ede863155" containerID="908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f" exitCode=0
Nov 03 11:19:31.773415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.773391    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"
Nov 03 11:19:31.773458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.773429    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" event={"ID":"406a01a7-6c91-4db3-a511-323ede863155","Type":"ContainerDied","Data":"908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f"}
Nov 03 11:19:31.773520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.773481    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb" event={"ID":"406a01a7-6c91-4db3-a511-323ede863155","Type":"ContainerDied","Data":"0c780793abfe4f3bd5dc10324557a3935bce072dafec3c0b45371f101f3b36f8"}
Nov 03 11:19:31.773520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.773515    5817 scope.go:122] "RemoveContainer" containerID="908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f"
Nov 03 11:19:31.786767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.786746    5817 scope.go:122] "RemoveContainer" containerID="a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae"
Nov 03 11:19:31.800488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.800471    5817 scope.go:122] "RemoveContainer" containerID="908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f"
Nov 03 11:19:31.800828 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:19:31.800809    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f\": container with ID starting with 908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f not found: ID does not exist" containerID="908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f"
Nov 03 11:19:31.800899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.800835    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f"} err="failed to get container status \"908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f\": rpc error: code = NotFound desc = could not find container \"908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f\": container with ID starting with 908b58e197b3bf0678609708c9697df15e56875d435d7244048a7bdac20f1c2f not found: ID does not exist"
Nov 03 11:19:31.800899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.800851    5817 scope.go:122] "RemoveContainer" containerID="a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae"
Nov 03 11:19:31.801127 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:19:31.801105    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae\": container with ID starting with a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae not found: ID does not exist" containerID="a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae"
Nov 03 11:19:31.801224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.801134    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae"} err="failed to get container status \"a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae\": rpc error: code = NotFound desc = could not find container \"a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae\": container with ID starting with a221e2dcc6f811029ef5621fd12bcf6e65e74b5da2976bee5e773cc8cbf8e6ae not found: ID does not exist"
Nov 03 11:19:31.811208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.811186    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"]
Nov 03 11:19:31.820970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.820946    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-tzvhb"]
Nov 03 11:19:31.913034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:31.912997    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406a01a7-6c91-4db3-a511-323ede863155" path="/var/lib/kubelet/pods/406a01a7-6c91-4db3-a511-323ede863155/volumes"
Nov 03 11:19:32.103765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:32.093219    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:32.786219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:32.786182    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" event={"ID":"aa54aa54-6c59-4169-a0f2-beb2ca32557b","Type":"ContainerStarted","Data":"ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f"}
Nov 03 11:19:32.831308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:32.829658    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" podStartSLOduration=94.829636699 podStartE2EDuration="1m34.829636699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:19:32.818058446 +0000 UTC m=+3003.023640498" watchObservedRunningTime="2026-11-03 11:19:32.829636699 +0000 UTC m=+3003.035218751"
Nov 03 11:19:32.864345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:32.848340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:33.445286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:33.445249    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:33.447105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:33.447073    5817 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 11:19:33.447222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:33.447131    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 11:19:33.711654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:33.709296    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:34.311612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:34.309001    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:34.445099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:34.445067    5817 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 11:19:34.445305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:34.445138    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 11:19:35.083602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:35.078678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:35.445788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:35.445757    5817 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 11:19:35.446210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:35.445822    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 11:19:35.789651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:35.785969    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:36.445784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:36.445748    5817 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 11:19:36.446205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:36.445815    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 11:19:36.528610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:36.526680    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"]
Nov 03 11:19:36.645441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:36.645403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:36.887648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:36.886944    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:37.440814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:37.440781    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:37.452548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:37.452512    5817 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 11:19:37.458695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:37.458653    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 11:19:38.043599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:38.042249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:38.445146 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:38.445110    5817 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 11:19:38.445322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:38.445173    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 11:19:38.714653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:38.713991    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:39.461598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:39.457797    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:39.537444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:39.534148    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:39.537444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:39.536106    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:39.560779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:39.560743    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:19:40.064498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:40.064458    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:40.800593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:40.795253    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:41.467610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:41.462541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:41.536867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:41.531712    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4" podUID="e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" containerName="payload-processing" containerID="cri-o://0116b0a7dbeab670717c57755588120707c00cf15c78dda485724befd897e7bb" gracePeriod=25
Nov 03 11:19:42.186601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:42.181318    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:42.863724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:42.863694    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:43.458167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:43.458083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:44.055687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:44.048099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:44.786308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:44.786223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:45.235141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:45.235103    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4" podUID="e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:19:45.439181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:45.439146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:46.065609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:46.064204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:46.727603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:46.726233    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:47.353606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:47.346105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:47.987776 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:47.987738    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:48.687449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:48.687390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:49.248545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:49.248242    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:49.878610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:49.875614    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:50.281448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:50.281411    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:19:50.578686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:50.566124    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:51.378419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:51.378383    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:52.698429 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:19:52.698396    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:19:52.761546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:52.761509    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:52.856747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:52.856649    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 11:19:52.858002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:52.857973    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:19:53.669600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:53.657825    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:54.140237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:54.140203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:54.275231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:54.275191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:54.410280 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:54.410188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:55.180278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:55.180224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:55.237423 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:55.237385    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4" podUID="e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:19:55.617387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:55.617303    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:55.810908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:55.810865    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:55.954189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:55.954163    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:55.963528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:55.963503    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"]
Nov 03 11:19:55.969060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:55.969038    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60ad03-chq4v"]
Nov 03 11:19:56.101244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:56.097816    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:56.503890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:56.500047    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:57.538599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:57.538508    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:57.910791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:57.910699    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73669a9-a186-485c-a0a9-09d68f9fb3a9" path="/var/lib/kubelet/pods/e73669a9-a186-485c-a0a9-09d68f9fb3a9/volumes"
Nov 03 11:19:58.285483 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:58.285445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:59.038525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:59.038492    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:19:59.808614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:19:59.805982    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:00.500596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:00.492987    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:01.255767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.255730    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:01.643664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.643635    5817 generic.go:356] "Generic (PLEG): container finished" podID="e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" containerID="0116b0a7dbeab670717c57755588120707c00cf15c78dda485724befd897e7bb" exitCode=0
Nov 03 11:20:01.644040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.643715    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4" event={"ID":"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09","Type":"ContainerDied","Data":"0116b0a7dbeab670717c57755588120707c00cf15c78dda485724befd897e7bb"}
Nov 03 11:20:01.830278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.830213    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:20:01.909355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.909325    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-plugins-config-volume\") pod \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\" (UID: \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\") "
Nov 03 11:20:01.909600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.909543    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-kube-api-access-ks976\" (UniqueName: \"kubernetes.io/projected/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-kube-api-access-ks976\") pod \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\" (UID: \"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09\") "
Nov 03 11:20:01.910146 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.910111    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-plugins-config-volume" pod "e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" (UID: "e832b6ad-efdc-418a-8a5f-c04bbf2d8b09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:20:01.910856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.910825    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:01.912509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:01.912485    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-kube-api-access-ks976" pod "e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" (UID: "e832b6ad-efdc-418a-8a5f-c04bbf2d8b09"). InnerVolumeSpecName "kube-api-access-ks976". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:02.013232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.013188    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks976\" (UniqueName: \"kubernetes.io/projected/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09-kube-api-access-ks976\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:02.188439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.188405    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:02.339359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.339319    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:02.472717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.472624    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:02.590831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.590790    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"]
Nov 03 11:20:02.591103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.591077    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy" containerID="cri-o://e463be0043bf19cbb832d2c21363cce5b7a11ecfc18d538d0d3064b31ae3777d" gracePeriod=30
Nov 03 11:20:02.656642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.656610    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"
Nov 03 11:20:02.657112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.656622    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4" event={"ID":"e832b6ad-efdc-418a-8a5f-c04bbf2d8b09","Type":"ContainerDied","Data":"a07d4930cf1c4aa5fef4ffae450154e2a4c3817d7d0f1943937f786f406e8a7b"}
Nov 03 11:20:02.657112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.656724    5817 scope.go:122] "RemoveContainer" containerID="0116b0a7dbeab670717c57755588120707c00cf15c78dda485724befd897e7bb"
Nov 03 11:20:02.658368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.658340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:02.828652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.826979    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"]
Nov 03 11:20:02.939221 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:02.939186    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60ad03-6bfcb8f847-545j4"]
Nov 03 11:20:03.329592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:03.316196    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:03.627531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:03.627425    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:03.764415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:03.764354    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:03.901257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:03.901163    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:03.909176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:03.909150    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" path="/var/lib/kubelet/pods/e832b6ad-efdc-418a-8a5f-c04bbf2d8b09/volumes"
Nov 03 11:20:04.048460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:04.048414    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:04.793601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:04.790893    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:05.268445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:05.268416    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"]
Nov 03 11:20:05.268674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:05.268652    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy" containerID="cri-o://b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481" gracePeriod=30
Nov 03 11:20:05.502610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:05.501718    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:06.166857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:06.166772    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:06.873540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:06.873506    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:07.506123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.506086    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:07.724006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.723967    5817 generic.go:356] "Generic (PLEG): container finished" podID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerID="e463be0043bf19cbb832d2c21363cce5b7a11ecfc18d538d0d3064b31ae3777d" exitCode=0
Nov 03 11:20:07.724140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.724039    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" event={"ID":"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64","Type":"ContainerDied","Data":"e463be0043bf19cbb832d2c21363cce5b7a11ecfc18d538d0d3064b31ae3777d"}
Nov 03 11:20:07.724224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.724147    5817 scope.go:122] "RemoveContainer" containerID="400e3df66e3806d2a15844a63c87f648c6fca1d31101706b6983758bcbd36c62"
Nov 03 11:20:07.935368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.935341    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:20:07.972423 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.972385    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-certs\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.972660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.972501    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istiod-ca-cert\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.972660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.972619    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-envoy\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.972794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.972710    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-certs" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:07.972794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.972728    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-socket\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.972910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.972811    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-podinfo\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.972910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.972844    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-token\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-token\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.973101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.973079    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istiod-ca-cert" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:20:07.973257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.973097    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-kube-api-access-ndgll\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-kube-api-access-ndgll\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.973257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.973242    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-credential-socket\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.973394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.973277    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-data\") pod \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\" (UID: \"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64\") "
Nov 03 11:20:07.973394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.973370    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-socket" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:07.973687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.973558    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-credential-socket" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:07.974335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.973969    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-data" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:07.974767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.974741    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:07.974894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.974770    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:07.975183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.975163    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:07.975183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.975186    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:07.975333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.975224    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:07.975978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.975758    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-token" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:07.976106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.976078    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-envoy" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:07.976415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.976392    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-kube-api-access-ndgll" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "kube-api-access-ndgll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:07.977271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:07.977250    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-podinfo" pod "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" (UID: "e1b5ae23-6e01-43e1-a5e4-9a87c50edd64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:20:08.077429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.077348    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:08.077429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.077381    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:08.077429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.077391    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:08.077429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.077400    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndgll\" (UniqueName: \"kubernetes.io/projected/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64-kube-api-access-ndgll\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:08.178789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.178700    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:08.328646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.325499    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:08.736535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.736506    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"
Nov 03 11:20:08.737142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.737103    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt" event={"ID":"e1b5ae23-6e01-43e1-a5e4-9a87c50edd64","Type":"ContainerDied","Data":"969de8d813507c87941e3fa9834621608309abacccc8d66062873b90042ec827"}
Nov 03 11:20:08.737286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.737178    5817 scope.go:122] "RemoveContainer" containerID="e463be0043bf19cbb832d2c21363cce5b7a11ecfc18d538d0d3064b31ae3777d"
Nov 03 11:20:08.767119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.767093    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"]
Nov 03 11:20:08.773451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.773431    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-60ad03-openshift-default-b4f95cf65-9jtpt"]
Nov 03 11:20:09.007644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:08.999477    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:09.799608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:09.799482    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:09.907707 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:09.907679    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" path="/var/lib/kubelet/pods/e1b5ae23-6e01-43e1-a5e4-9a87c50edd64/volumes"
Nov 03 11:20:10.408607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.408520    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:10.664607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.664552    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 11:20:10.763143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.763106    5817 generic.go:356] "Generic (PLEG): container finished" podID="975edffc-3c93-4c22-a72b-161989a9b772" containerID="b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481" exitCode=0
Nov 03 11:20:10.763359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.763161    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"
Nov 03 11:20:10.763359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.763213    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" event={"ID":"975edffc-3c93-4c22-a72b-161989a9b772","Type":"ContainerDied","Data":"b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481"}
Nov 03 11:20:10.763359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.763290    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq" event={"ID":"975edffc-3c93-4c22-a72b-161989a9b772","Type":"ContainerDied","Data":"a107fbf1952fb32fb71460cdc905a22fea20618c9340a229c70835bcc1e294d8"}
Nov 03 11:20:10.763359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.763341    5817 scope.go:122] "RemoveContainer" containerID="b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481"
Nov 03 11:20:10.779538 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.779500    5817 scope.go:122] "RemoveContainer" containerID="b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481"
Nov 03 11:20:10.779894 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:20:10.779870    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481\": container with ID starting with b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481 not found: ID does not exist" containerID="b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481"
Nov 03 11:20:10.779988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.779902    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481"} err="failed to get container status \"b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481\": rpc error: code = NotFound desc = could not find container \"b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481\": container with ID starting with b4697e18f2ebd86728463b9c673575612ea412794248cdceeccd417d70bfe481 not found: ID does not exist"
Nov 03 11:20:10.834558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.834515    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-certs\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.834558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.834565    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-socket\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.835161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.834597    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/975edffc-3c93-4c22-a72b-161989a9b772-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975edffc-3c93-4c22-a72b-161989a9b772-istio-podinfo\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.835161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.834845    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-socket" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:10.835161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.834854    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-certs" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:10.835161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.834913    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975edffc-3c93-4c22-a72b-161989a9b772-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975edffc-3c93-4c22-a72b-161989a9b772-istiod-ca-cert\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.835161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.834976    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-credential-socket\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.835161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.835055    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-kube-api-access-k8s6k\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-kube-api-access-k8s6k\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.835654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.835198    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-istio-token\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-istio-token\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.835654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.835301    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-envoy\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.835654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.835334    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-data\") pod \"975edffc-3c93-4c22-a72b-161989a9b772\" (UID: \"975edffc-3c93-4c22-a72b-161989a9b772\") "
Nov 03 11:20:10.835654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.835391    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-credential-socket" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:10.836560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.836520    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975edffc-3c93-4c22-a72b-161989a9b772-istiod-ca-cert" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:20:10.836745 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.836616    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:10.836745 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.836638    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:10.836745 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.836653    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:10.837048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.836934    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-data" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:10.840232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.840194    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/975edffc-3c93-4c22-a72b-161989a9b772-istio-podinfo" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:20:10.840514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.840429    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-envoy" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:10.840514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.840474    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-kube-api-access-k8s6k" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "kube-api-access-k8s6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:10.840698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.840664    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-istio-token" pod "975edffc-3c93-4c22-a72b-161989a9b772" (UID: "975edffc-3c93-4c22-a72b-161989a9b772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:10.938694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.938665    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:10.938694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.938696    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:10.938873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.938710    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975edffc-3c93-4c22-a72b-161989a9b772-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:10.938873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.938725    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975edffc-3c93-4c22-a72b-161989a9b772-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:10.938873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.938738    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975edffc-3c93-4c22-a72b-161989a9b772-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:10.938873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:10.938752    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8s6k\" (UniqueName: \"kubernetes.io/projected/975edffc-3c93-4c22-a72b-161989a9b772-kube-api-access-k8s6k\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:11.089598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:11.088243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:11.163598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:11.159624    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"]
Nov 03 11:20:11.169850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:11.169724    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-vhlpq"]
Nov 03 11:20:11.790347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:11.790312    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:11.907894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:11.907858    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975edffc-3c93-4c22-a72b-161989a9b772" path="/var/lib/kubelet/pods/975edffc-3c93-4c22-a72b-161989a9b772/volumes"
Nov 03 11:20:12.404594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:12.399315    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:13.301591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:13.300701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:13.961843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:13.961639    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:14.486012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:14.485980    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:14.632599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:14.632547    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:14.774015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:14.773933    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:15.355146 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:15.355109    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:15.972600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:15.968371    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:16.633712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:16.633676    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:17.363300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:17.363252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:18.078600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:18.074988    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:18.616467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:18.616431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:18.776597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:18.769617    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:19.396594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:19.396423    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:20.074240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:20.074153    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:20.729300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:20.729265    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:21.444559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:21.444524    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:22.077863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:22.077827    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:22.419259 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:20:22.419181    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:20:22.691598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:22.682297    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:23.334945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:23.334894    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:23.957600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:23.956484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:24.644318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:24.642000    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:25.381621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:25.373131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:25.985640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:25.983203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:26.597600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:26.597103    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:27.308732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:27.308701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:27.923961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:27.923892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:28.591780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:28.591730    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:29.347844 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:29.347808    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:29.957606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:29.951072    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:30.648195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:30.648158    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:31.230403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:31.230367    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:31.889165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:31.889093    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:32.553628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:32.549723    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:33.171600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:33.170894    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:33.959597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:33.941935    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:34.545199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:34.545144    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:34.947464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:34.947431    5817 scope.go:122] "RemoveContainer" containerID="9980ccd714c08f3af5875965ee1206d7058bb30b953ea0e57263bbf6ef83a1a7"
Nov 03 11:20:34.963896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:34.963864    5817 scope.go:122] "RemoveContainer" containerID="cadd7d99abcf9273708d70fceec95c143ba5d5966e2d4adeea911567fbc7e140"
Nov 03 11:20:34.975834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:34.975808    5817 scope.go:122] "RemoveContainer" containerID="d213f819ad91b4303d082a4c85b23887c4b7e5333391192ae571b65713ab8408"
Nov 03 11:20:34.986829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:34.986807    5817 scope.go:122] "RemoveContainer" containerID="5ab6d9675bbce4cb9e1730e6a94362e37aed5a07f91d4db01a0f9141c7d3a3b4"
Nov 03 11:20:35.197220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:35.197182    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:35.957220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:35.957126    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:36.045597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:36.045256    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"]
Nov 03 11:20:36.054011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:36.053985    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0476e0bd-lcsmf"]
Nov 03 11:20:36.694597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:36.687023    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:37.372613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:37.370046    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:38.007116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:38.007076    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f197cb36-4238-451d-bb21-d06d76a18460" path="/var/lib/kubelet/pods/f197cb36-4238-451d-bb21-d06d76a18460/volumes"
Nov 03 11:20:38.400867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:38.400593    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:39.283077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:39.283028    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:40.112598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:40.111746    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:41.087714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:41.087527    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:41.776858 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:41.776757    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:42.476615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:42.464225    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:43.387598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:43.382406    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:44.006600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:44.005239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:44.721487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:44.721450    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:45.475150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:45.475112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:46.041756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:46.041717    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"]
Nov 03 11:20:46.049827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:46.049803    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a224c-l8rqg"]
Nov 03 11:20:46.140278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:46.140239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:46.767599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:46.760734    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:47.407596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:47.403809    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:47.906618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:47.906533    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cac2d4-6200-4637-bbd8-28d850759492" path="/var/lib/kubelet/pods/81cac2d4-6200-4637-bbd8-28d850759492/volumes"
Nov 03 11:20:48.064684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:48.064645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:48.767046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:48.766997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:49.391595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:49.380299    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:50.083103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:50.082390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:50.655200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:50.655155    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:51.302607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:51.301710    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:51.863597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:51.862353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:52.420762 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:20:52.420712    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:20:52.516604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:52.507947    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:53.232857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:53.228411    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:53.861122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:53.861091    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:54.203039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:54.202999    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:54.723594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:54.715301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:55.345524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:55.341183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:55.977254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:55.976823    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:56.774732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:56.774697    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:57.473695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:57.472557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:58.276226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:58.276173    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:58.982142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:58.982100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:20:59.697174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:20:59.697134    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:00.499597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:00.499260    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:01.198593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:01.197664    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:01.987006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:01.986970    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:02.686617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:02.677705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:03.367597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:03.364278    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:04.045620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:04.045589    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:04.739380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:04.739351    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:05.038830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.038731    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"]
Nov 03 11:21:05.046167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.046138    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72bc277-mj6xl"]
Nov 03 11:21:05.083690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.083654    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"]
Nov 03 11:21:05.084624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084607    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy"
Nov 03 11:21:05.084696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084625    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy"
Nov 03 11:21:05.084696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084651    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy"
Nov 03 11:21:05.084696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084656    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy"
Nov 03 11:21:05.084696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084668    5817 cpu_manager.go:414] "containerMap: removing container" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy"
Nov 03 11:21:05.084696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084673    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406a01a7-6c91-4db3-a511-323ede863155" containerName="istio-proxy"
Nov 03 11:21:05.084696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084686    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" containerName="payload-processing"
Nov 03 11:21:05.084696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084690    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" containerName="payload-processing"
Nov 03 11:21:05.084931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084717    5817 cpu_manager.go:414] "containerMap: removing container" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy"
Nov 03 11:21:05.084931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084721    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy"
Nov 03 11:21:05.084931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084865    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="975edffc-3c93-4c22-a72b-161989a9b772" containerName="istio-proxy"
Nov 03 11:21:05.084931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084880    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy"
Nov 03 11:21:05.084931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084888    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b5ae23-6e01-43e1-a5e4-9a87c50edd64" containerName="istio-proxy"
Nov 03 11:21:05.084931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.084896    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e832b6ad-efdc-418a-8a5f-c04bbf2d8b09" containerName="payload-processing"
Nov 03 11:21:05.094868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.094840    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.096394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.096360    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"]
Nov 03 11:21:05.098200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.098174    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:21:05.098375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.098360    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:21:05.246427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.246388    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.246638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.246622    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.246696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.246677    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmjb\" (UniqueName: \"kubernetes.io/projected/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-kube-api-access-ktmjb\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.337533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.337226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:05.354613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.348453    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.355137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.355047    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmjb\" (UniqueName: \"kubernetes.io/projected/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-kube-api-access-ktmjb\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.372410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.372374    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.373128 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.367954    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmjb\" (UniqueName: \"kubernetes.io/projected/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-kube-api-access-ktmjb\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.373310 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.355128    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.374626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.374560    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d851a1a-5c4bf\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.414778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.414731    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:05.721469 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:21:05.721435    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e74348c_5720_4bc9_ad8b_5cf35a7cc2da.slice/crio-71a7c9b9656dc266f4af72aded3b45d98a34e09d05d76eea55e4c60dd7e25212 WatchSource:0}: Error finding container 71a7c9b9656dc266f4af72aded3b45d98a34e09d05d76eea55e4c60dd7e25212: Status 404 returned error can't find the container with id 71a7c9b9656dc266f4af72aded3b45d98a34e09d05d76eea55e4c60dd7e25212
Nov 03 11:21:05.722047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.722007    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"]
Nov 03 11:21:05.907905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.907873    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b210b14-173e-44af-a9c1-e186d9572370" path="/var/lib/kubelet/pods/2b210b14-173e-44af-a9c1-e186d9572370/volumes"
Nov 03 11:21:05.964155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:05.964116    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf" event={"ID":"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da","Type":"ContainerStarted","Data":"71a7c9b9656dc266f4af72aded3b45d98a34e09d05d76eea55e4c60dd7e25212"}
Nov 03 11:21:06.078241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:06.078150    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:06.355285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:06.355192    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:06.677648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:06.677336    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:06.717593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:06.717179    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:21:06.975787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:06.975758    5817 generic.go:356] "Generic (PLEG): container finished" podID="1e74348c-5720-4bc9-ad8b-5cf35a7cc2da" containerID="83102fa181170c0e8c3e9cf946bdb00792ae0e62987bbb017142e647eefd6dfe" exitCode=0
Nov 03 11:21:06.975936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:06.975841    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf" event={"ID":"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da","Type":"ContainerDied","Data":"83102fa181170c0e8c3e9cf946bdb00792ae0e62987bbb017142e647eefd6dfe"}
Nov 03 11:21:07.110171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:07.110137    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:07.387715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:07.384827    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:08.060598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.060200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:08.356240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.356123    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:08.581621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.581590    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:08.684330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.684291    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-openshift-ca\") pod \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") "
Nov 03 11:21:08.684935 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.684906    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-xks-ca\") pod \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") "
Nov 03 11:21:08.685077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.685027    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-kube-api-access-ktmjb\" (UniqueName: \"kubernetes.io/projected/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-kube-api-access-ktmjb\") pod \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\" (UID: \"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da\") "
Nov 03 11:21:08.686276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.684730    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-openshift-ca" pod "1e74348c-5720-4bc9-ad8b-5cf35a7cc2da" (UID: "1e74348c-5720-4bc9-ad8b-5cf35a7cc2da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:08.689869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.689828    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-kube-api-access-ktmjb" pod "1e74348c-5720-4bc9-ad8b-5cf35a7cc2da" (UID: "1e74348c-5720-4bc9-ad8b-5cf35a7cc2da"). InnerVolumeSpecName "kube-api-access-ktmjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:08.689983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.689915    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-xks-ca" pod "1e74348c-5720-4bc9-ad8b-5cf35a7cc2da" (UID: "1e74348c-5720-4bc9-ad8b-5cf35a7cc2da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:08.788515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.788478    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:08.788678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.788518    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktmjb\" (UniqueName: \"kubernetes.io/projected/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-kube-api-access-ktmjb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:08.788678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:08.788537    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:09.028952 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:09.028869    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"
Nov 03 11:21:09.028952 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:09.028903    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf" event={"ID":"1e74348c-5720-4bc9-ad8b-5cf35a7cc2da","Type":"ContainerDied","Data":"71a7c9b9656dc266f4af72aded3b45d98a34e09d05d76eea55e4c60dd7e25212"}
Nov 03 11:21:09.029189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:09.028953    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a7c9b9656dc266f4af72aded3b45d98a34e09d05d76eea55e4c60dd7e25212"
Nov 03 11:21:09.094463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:09.088380    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:09.757608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:09.751050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:10.539512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:10.539479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:11.211333 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:11.211246    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:11.886526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:11.886494    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:12.502606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.486649    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:12.837131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.837044    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"]
Nov 03 11:21:12.838491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.838465    5817 cpu_manager.go:414] "containerMap: removing container" podUID="1e74348c-5720-4bc9-ad8b-5cf35a7cc2da" containerName="revoke-keys"
Nov 03 11:21:12.838491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.838490    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e74348c-5720-4bc9-ad8b-5cf35a7cc2da" containerName="revoke-keys"
Nov 03 11:21:12.838746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.838730    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e74348c-5720-4bc9-ad8b-5cf35a7cc2da" containerName="revoke-keys"
Nov 03 11:21:12.845923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.845899    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.877671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.877633    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"]
Nov 03 11:21:12.989982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.989944    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.990168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.990034    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.990168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.990073    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.990168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.990095    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.990316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.990209    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzsr\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-kube-api-access-gtzsr\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.990369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.990358    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.990441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.990425    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908808c-9d5c-4748-a226-c73289d2b7ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.990535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.990521    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:12.990704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:12.990689    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.091948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.091877    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.091948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.091917    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.091948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.091934    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.092255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.091980    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzsr\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-kube-api-access-gtzsr\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.092255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.092098    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.092255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.092162    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908808c-9d5c-4748-a226-c73289d2b7ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.092528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.092255    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.092528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.092333    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.092528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.092432    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.092528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.092501    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.092941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.092858    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.093397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.093365    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.093533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.093396    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.093533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.093466    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908808c-9d5c-4748-a226-c73289d2b7ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.094863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.094841    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.095005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.094984    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.099500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.099476    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.099921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.099904    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzsr\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-kube-api-access-gtzsr\") pod \"maas-default-gateway-openshift-default-6cc458d887-7hrjn\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.178607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.172279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:13.207609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.205811    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:13.526314 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:21:13.526281    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0908808c_9d5c_4748_a226_c73289d2b7ed.slice/crio-fa45226b1b6e4772163693451925ad9de0f38a0b08eeef845ec4ea86a03c81a6 WatchSource:0}: Error finding container fa45226b1b6e4772163693451925ad9de0f38a0b08eeef845ec4ea86a03c81a6: Status 404 returned error can't find the container with id fa45226b1b6e4772163693451925ad9de0f38a0b08eeef845ec4ea86a03c81a6
Nov 03 11:21:13.526804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.526676    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"]
Nov 03 11:21:13.530563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.530526    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:21:13.530662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.530631    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:21:13.530710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.530675    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:21:13.945244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:13.926121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:14.094997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:14.094955    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" event={"ID":"0908808c-9d5c-4748-a226-c73289d2b7ed","Type":"ContainerStarted","Data":"9148b9b95108de1fd3317e6c1bdd88def9cac2bf61e9a472be8bff59347be07f"}
Nov 03 11:21:14.095189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:14.095018    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" event={"ID":"0908808c-9d5c-4748-a226-c73289d2b7ed","Type":"ContainerStarted","Data":"fa45226b1b6e4772163693451925ad9de0f38a0b08eeef845ec4ea86a03c81a6"}
Nov 03 11:21:14.119539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:14.119479    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podStartSLOduration=2.119458473 podStartE2EDuration="2.119458473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:21:14.112862589 +0000 UTC m=+3104.318444643" watchObservedRunningTime="2026-11-03 11:21:14.119458473 +0000 UTC m=+3104.325040525"
Nov 03 11:21:14.207303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:14.207210    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:14.208901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:14.208873    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 11:21:14.209057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:14.208926    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 11:21:14.728599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:14.725403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:15.027510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:15.027411    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"]
Nov 03 11:21:15.035687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:15.035656    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed0d818-4jljw"]
Nov 03 11:21:15.217431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:15.216975    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 11:21:15.217431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:15.217038    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 11:21:15.444591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:15.442761    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:15.802736 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:15.800989    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:16.163127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.163038    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3009ca-a4e6-46f3-99fa-9da536815739" path="/var/lib/kubelet/pods/af3009ca-a4e6-46f3-99fa-9da536815739/volumes"
Nov 03 11:21:16.219598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.219003    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 11:21:16.219598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.219076    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 11:21:16.360102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.360069    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"]
Nov 03 11:21:16.365510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.365488    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.370552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.370529    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:21:16.370743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.370728    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:21:16.373677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.373643    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"]
Nov 03 11:21:16.428708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.427600    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxs2\" (UniqueName: \"kubernetes.io/projected/96c84c58-b971-4029-bf01-734c5bc1cab6-kube-api-access-2jxs2\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.428708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.427864    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.428708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.427895    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.530431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.529424    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxs2\" (UniqueName: \"kubernetes.io/projected/96c84c58-b971-4029-bf01-734c5bc1cab6-kube-api-access-2jxs2\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.530431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.529637    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.530431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.529665    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.530777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.530738    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.534063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.534032    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.541276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.541247    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxs2\" (UniqueName: \"kubernetes.io/projected/96c84c58-b971-4029-bf01-734c5bc1cab6-kube-api-access-2jxs2\") pod \"maas-api-revoke-sub-6eedce5f-8g4cp\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:16.559343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.559316    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:16.725675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:16.725054    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:17.071727 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:21:17.071694    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c84c58_b971_4029_bf01_734c5bc1cab6.slice/crio-311cd792847c51ae957aa34406777f6f092f2340b927155ac87971b874e51742 WatchSource:0}: Error finding container 311cd792847c51ae957aa34406777f6f092f2340b927155ac87971b874e51742: Status 404 returned error can't find the container with id 311cd792847c51ae957aa34406777f6f092f2340b927155ac87971b874e51742
Nov 03 11:21:17.073125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:17.073105    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"]
Nov 03 11:21:17.215596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:17.207382    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp" event={"ID":"96c84c58-b971-4029-bf01-734c5bc1cab6","Type":"ContainerStarted","Data":"311cd792847c51ae957aa34406777f6f092f2340b927155ac87971b874e51742"}
Nov 03 11:21:17.215596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:17.208187    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 11:21:17.215596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:17.208240    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 11:21:17.528394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:17.524712    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:18.215605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:18.210862    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 11:21:18.215605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:18.210922    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 11:21:18.232625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:18.229158    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp" event={"ID":"96c84c58-b971-4029-bf01-734c5bc1cab6","Type":"ContainerStarted","Data":"f7c755e84aa0d00fa6353b919f5ac6d1be8dcf9c4dc82273b38f57963e52a83a"}
Nov 03 11:21:18.232625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:18.231483    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:18.259944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:18.259765    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp" podStartSLOduration=2.2597458 podStartE2EDuration="2.2597458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:21:18.246418136 +0000 UTC m=+3108.452000186" watchObservedRunningTime="2026-11-03 11:21:18.2597458 +0000 UTC m=+3108.465327846"
Nov 03 11:21:18.664906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:18.664870    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:18.993609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:18.992680    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:19.219683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:19.218072    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 11:21:19.219683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:19.218165    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 11:21:19.242312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:19.242283    5817 generic.go:356] "Generic (PLEG): container finished" podID="96c84c58-b971-4029-bf01-734c5bc1cab6" containerID="f7c755e84aa0d00fa6353b919f5ac6d1be8dcf9c4dc82273b38f57963e52a83a" exitCode=0
Nov 03 11:21:19.242492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:19.242382    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp" event={"ID":"96c84c58-b971-4029-bf01-734c5bc1cab6","Type":"ContainerDied","Data":"f7c755e84aa0d00fa6353b919f5ac6d1be8dcf9c4dc82273b38f57963e52a83a"}
Nov 03 11:21:19.659488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:19.659453    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:20.207297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.207262    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 11:21:20.207496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.207320    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 11:21:20.530934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.530834    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:20.566725 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.566686    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:20.667379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.667349    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-openshift-ca\") pod \"96c84c58-b971-4029-bf01-734c5bc1cab6\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") "
Nov 03 11:21:20.667699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.667634    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-xks-ca\") pod \"96c84c58-b971-4029-bf01-734c5bc1cab6\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") "
Nov 03 11:21:20.667797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.667736    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c84c58-b971-4029-bf01-734c5bc1cab6-kube-api-access-2jxs2\" (UniqueName: \"kubernetes.io/projected/96c84c58-b971-4029-bf01-734c5bc1cab6-kube-api-access-2jxs2\") pod \"96c84c58-b971-4029-bf01-734c5bc1cab6\" (UID: \"96c84c58-b971-4029-bf01-734c5bc1cab6\") "
Nov 03 11:21:20.667895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.667843    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-openshift-ca" pod "96c84c58-b971-4029-bf01-734c5bc1cab6" (UID: "96c84c58-b971-4029-bf01-734c5bc1cab6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:20.668336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.668312    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:20.670712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.670685    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-xks-ca" pod "96c84c58-b971-4029-bf01-734c5bc1cab6" (UID: "96c84c58-b971-4029-bf01-734c5bc1cab6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:20.670852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.670826    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c84c58-b971-4029-bf01-734c5bc1cab6-kube-api-access-2jxs2" pod "96c84c58-b971-4029-bf01-734c5bc1cab6" (UID: "96c84c58-b971-4029-bf01-734c5bc1cab6"). InnerVolumeSpecName "kube-api-access-2jxs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:20.769998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.769973    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jxs2\" (UniqueName: \"kubernetes.io/projected/96c84c58-b971-4029-bf01-734c5bc1cab6-kube-api-access-2jxs2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:20.769998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:20.770000    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c84c58-b971-4029-bf01-734c5bc1cab6-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:21.206615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:21.206544    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 11:21:21.206840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:21.206618    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 11:21:21.285675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:21.283405    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp" event={"ID":"96c84c58-b971-4029-bf01-734c5bc1cab6","Type":"ContainerDied","Data":"311cd792847c51ae957aa34406777f6f092f2340b927155ac87971b874e51742"}
Nov 03 11:21:21.285675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:21.283469    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="311cd792847c51ae957aa34406777f6f092f2340b927155ac87971b874e51742"
Nov 03 11:21:21.285675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:21.283537    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"
Nov 03 11:21:21.345604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:21.343204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:22.173991 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:22.173953    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:22.422001 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:21:22.421967    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:21:22.886218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:22.886184    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:22.886703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:22.886676    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:22.888332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:22.888153    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:21:22.970603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:22.969131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:23.633130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:23.633095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:24.379598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:24.372763    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:25.102371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:25.102295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:25.764378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:25.764340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:26.563602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:26.560479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:27.040068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.040030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:27.360355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.360276    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"]
Nov 03 11:21:27.361286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.361267    5817 cpu_manager.go:414] "containerMap: removing container" podUID="96c84c58-b971-4029-bf01-734c5bc1cab6" containerName="revoke-keys"
Nov 03 11:21:27.361352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.361287    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c84c58-b971-4029-bf01-734c5bc1cab6" containerName="revoke-keys"
Nov 03 11:21:27.361498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.361489    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c84c58-b971-4029-bf01-734c5bc1cab6" containerName="revoke-keys"
Nov 03 11:21:27.365618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.365558    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.372394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.372362    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:21:27.372527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.372413    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:21:27.372667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.372651    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"]
Nov 03 11:21:27.507684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.507596    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.530076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.526028    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.535020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.530617    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlkd8\" (UniqueName: \"kubernetes.io/projected/f1bfa725-687b-4bbc-8686-ff41c579d5d4-kube-api-access-vlkd8\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.535903 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.526539    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:27.637042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.636949    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.637552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.637052    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlkd8\" (UniqueName: \"kubernetes.io/projected/f1bfa725-687b-4bbc-8686-ff41c579d5d4-kube-api-access-vlkd8\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.637552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.637314    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.637851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.637827    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.640008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.639986    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.645740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.645720    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlkd8\" (UniqueName: \"kubernetes.io/projected/f1bfa725-687b-4bbc-8686-ff41c579d5d4-kube-api-access-vlkd8\") pod \"maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.695917 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.695895    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:27.767708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.767673    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:27.985406 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:21:27.985379    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bfa725_687b_4bbc_8686_ff41c579d5d4.slice/crio-79429b15d8aa02e111516788910711556b90461443f8ee53477311511dfa55d0 WatchSource:0}: Error finding container 79429b15d8aa02e111516788910711556b90461443f8ee53477311511dfa55d0: Status 404 returned error can't find the container with id 79429b15d8aa02e111516788910711556b90461443f8ee53477311511dfa55d0
Nov 03 11:21:27.986786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:27.986755    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"]
Nov 03 11:21:28.281646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:28.272338    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:28.571823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:28.571728    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h" event={"ID":"f1bfa725-687b-4bbc-8686-ff41c579d5d4","Type":"ContainerStarted","Data":"b1215fb51ef829e7120d10b0d7f1b76547d9cd7fdddde475a40ea763c7b576b5"}
Nov 03 11:21:28.571823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:28.571790    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h" event={"ID":"f1bfa725-687b-4bbc-8686-ff41c579d5d4","Type":"ContainerStarted","Data":"79429b15d8aa02e111516788910711556b90461443f8ee53477311511dfa55d0"}
Nov 03 11:21:28.587675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:28.587626    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h" podStartSLOduration=1.587609773 podStartE2EDuration="1.587609773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:21:28.586329839 +0000 UTC m=+3118.791911904" watchObservedRunningTime="2026-11-03 11:21:28.587609773 +0000 UTC m=+3118.793191815"
Nov 03 11:21:29.028064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:29.028031    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"]
Nov 03 11:21:29.033491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:29.033467    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f58ea23-rh677"]
Nov 03 11:21:29.109910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:29.108230    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:29.585009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:29.584980    5817 generic.go:356] "Generic (PLEG): container finished" podID="f1bfa725-687b-4bbc-8686-ff41c579d5d4" containerID="b1215fb51ef829e7120d10b0d7f1b76547d9cd7fdddde475a40ea763c7b576b5" exitCode=0
Nov 03 11:21:29.585191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:29.585048    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h" event={"ID":"f1bfa725-687b-4bbc-8686-ff41c579d5d4","Type":"ContainerDied","Data":"b1215fb51ef829e7120d10b0d7f1b76547d9cd7fdddde475a40ea763c7b576b5"}
Nov 03 11:21:29.872219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:29.872073    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:29.980409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:29.979312    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4588367e-2641-4dc2-a783-99248c2de54c" path="/var/lib/kubelet/pods/4588367e-2641-4dc2-a783-99248c2de54c/volumes"
Nov 03 11:21:30.569063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.568295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:30.846626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.846541    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:30.867926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.867893    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bfa725-687b-4bbc-8686-ff41c579d5d4-kube-api-access-vlkd8\" (UniqueName: \"kubernetes.io/projected/f1bfa725-687b-4bbc-8686-ff41c579d5d4-kube-api-access-vlkd8\") pod \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") "
Nov 03 11:21:30.868338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.868316    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-openshift-ca\") pod \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") "
Nov 03 11:21:30.868459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.868358    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-xks-ca\") pod \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\" (UID: \"f1bfa725-687b-4bbc-8686-ff41c579d5d4\") "
Nov 03 11:21:30.868748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.868724    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-openshift-ca" pod "f1bfa725-687b-4bbc-8686-ff41c579d5d4" (UID: "f1bfa725-687b-4bbc-8686-ff41c579d5d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:30.869204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.869182    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:30.870717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.870689    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-xks-ca" pod "f1bfa725-687b-4bbc-8686-ff41c579d5d4" (UID: "f1bfa725-687b-4bbc-8686-ff41c579d5d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:30.871172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.871150    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bfa725-687b-4bbc-8686-ff41c579d5d4-kube-api-access-vlkd8" pod "f1bfa725-687b-4bbc-8686-ff41c579d5d4" (UID: "f1bfa725-687b-4bbc-8686-ff41c579d5d4"). InnerVolumeSpecName "kube-api-access-vlkd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:30.970438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.970410    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlkd8\" (UniqueName: \"kubernetes.io/projected/f1bfa725-687b-4bbc-8686-ff41c579d5d4-kube-api-access-vlkd8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:30.970438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:30.970435    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfa725-687b-4bbc-8686-ff41c579d5d4-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:31.253754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.250071    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:31.574863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.574675    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"]
Nov 03 11:21:31.575730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.575712    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f1bfa725-687b-4bbc-8686-ff41c579d5d4" containerName="revoke-keys"
Nov 03 11:21:31.575730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.575731    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bfa725-687b-4bbc-8686-ff41c579d5d4" containerName="revoke-keys"
Nov 03 11:21:31.575942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.575931    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bfa725-687b-4bbc-8686-ff41c579d5d4" containerName="revoke-keys"
Nov 03 11:21:31.580097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.580075    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.586936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.586912    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"]
Nov 03 11:21:31.657506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.657472    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"
Nov 03 11:21:31.657506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.657491    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h" event={"ID":"f1bfa725-687b-4bbc-8686-ff41c579d5d4","Type":"ContainerDied","Data":"79429b15d8aa02e111516788910711556b90461443f8ee53477311511dfa55d0"}
Nov 03 11:21:31.657742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.657552    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79429b15d8aa02e111516788910711556b90461443f8ee53477311511dfa55d0"
Nov 03 11:21:31.683627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.683602    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.683891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.683868    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.683982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.683919    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmqf8\" (UniqueName: \"kubernetes.io/projected/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-kube-api-access-cmqf8\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.786080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.786050    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.786281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.786250    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.786281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.786279    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmqf8\" (UniqueName: \"kubernetes.io/projected/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-kube-api-access-cmqf8\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.786789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.786764    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.788754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.788728    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.794281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.794260    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmqf8\" (UniqueName: \"kubernetes.io/projected/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-kube-api-access-cmqf8\") pod \"maas-api-revoke-sub-fb0cff11-l4h42\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:31.906167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:31.906098    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:32.216964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:32.216930    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:32.264537 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:21:32.264501    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013cd32c_45d0_4685_a9cd_bbeb9ab78bea.slice/crio-19f0e913b424b2a52bc66ebec96dcafd22513584f4ea583c8b4239d834e35069 WatchSource:0}: Error finding container 19f0e913b424b2a52bc66ebec96dcafd22513584f4ea583c8b4239d834e35069: Status 404 returned error can't find the container with id 19f0e913b424b2a52bc66ebec96dcafd22513584f4ea583c8b4239d834e35069
Nov 03 11:21:32.265074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:32.265047    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"]
Nov 03 11:21:32.669139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:32.669108    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42" event={"ID":"013cd32c-45d0-4685-a9cd-bbeb9ab78bea","Type":"ContainerStarted","Data":"19f0e913b424b2a52bc66ebec96dcafd22513584f4ea583c8b4239d834e35069"}
Nov 03 11:21:32.866159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:32.866015    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:33.579706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:33.571254    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:33.682264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:33.682231    5817 generic.go:356] "Generic (PLEG): container finished" podID="013cd32c-45d0-4685-a9cd-bbeb9ab78bea" containerID="43dec831b8aabfdcbc3741cf7c1fc407d8d993e260732c6c680bd7aed6f26175" exitCode=0
Nov 03 11:21:33.682676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:33.682368    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42" event={"ID":"013cd32c-45d0-4685-a9cd-bbeb9ab78bea","Type":"ContainerDied","Data":"43dec831b8aabfdcbc3741cf7c1fc407d8d993e260732c6c680bd7aed6f26175"}
Nov 03 11:21:34.157050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.157018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:34.333619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.332598    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:34.893602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.893547    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:34.958822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.958737    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-kube-api-access-cmqf8\" (UniqueName: \"kubernetes.io/projected/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-kube-api-access-cmqf8\") pod \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") "
Nov 03 11:21:34.958995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.958894    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-openshift-ca\") pod \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") "
Nov 03 11:21:34.958995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.958934    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-xks-ca\") pod \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\" (UID: \"013cd32c-45d0-4685-a9cd-bbeb9ab78bea\") "
Nov 03 11:21:34.959327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.959299    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-openshift-ca" pod "013cd32c-45d0-4685-a9cd-bbeb9ab78bea" (UID: "013cd32c-45d0-4685-a9cd-bbeb9ab78bea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:34.960183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.960161    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:34.967199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.961916    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-xks-ca" pod "013cd32c-45d0-4685-a9cd-bbeb9ab78bea" (UID: "013cd32c-45d0-4685-a9cd-bbeb9ab78bea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:34.967199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:34.962198    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-kube-api-access-cmqf8" pod "013cd32c-45d0-4685-a9cd-bbeb9ab78bea" (UID: "013cd32c-45d0-4685-a9cd-bbeb9ab78bea"). InnerVolumeSpecName "kube-api-access-cmqf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:35.051620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:35.034478    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:35.076625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:35.076285    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmqf8\" (UniqueName: \"kubernetes.io/projected/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-kube-api-access-cmqf8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:35.076625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:35.076329    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013cd32c-45d0-4685-a9cd-bbeb9ab78bea-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:35.092649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:35.091267    5817 scope.go:122] "RemoveContainer" containerID="1e75fb415d1a00379337a0320319c7a7fd04065e301f431c1a2e5a6aa05261a1"
Nov 03 11:21:35.693595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:35.690012    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:35.754610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:35.754057    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42" event={"ID":"013cd32c-45d0-4685-a9cd-bbeb9ab78bea","Type":"ContainerDied","Data":"19f0e913b424b2a52bc66ebec96dcafd22513584f4ea583c8b4239d834e35069"}
Nov 03 11:21:35.754610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:35.754123    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f0e913b424b2a52bc66ebec96dcafd22513584f4ea583c8b4239d834e35069"
Nov 03 11:21:35.754610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:35.754196    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"
Nov 03 11:21:36.284596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:36.277035    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:37.011597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:37.010625    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:37.470154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:37.470121    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"]
Nov 03 11:21:37.580699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:37.580657    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:37.710078 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:37.710036    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:37.878648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:37.871323    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:38.587605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:38.587207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:39.028475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:39.028441    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"]
Nov 03 11:21:39.037229 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:39.037204    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beeaacc7-g25kc"]
Nov 03 11:21:39.240289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:39.240238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:39.864618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:39.864559    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:39.951730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:39.951699    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ac2f9f-936f-4925-b796-03c133b405ec" path="/var/lib/kubelet/pods/d4ac2f9f-936f-4925-b796-03c133b405ec/volumes"
Nov 03 11:21:40.485622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:40.483201    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:41.104627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:41.093914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:41.688508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:41.688407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:42.408596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.401426    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:42.471682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.471638    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj" podUID="3d5d9e99-aa64-4a6d-a11d-1617ef5e643e" containerName="payload-processing" containerID="cri-o://e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487" gracePeriod=25
Nov 03 11:21:42.672794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.672703    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:42.796631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.795988    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"]
Nov 03 11:21:42.798063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.798016    5817 cpu_manager.go:414] "containerMap: removing container" podUID="013cd32c-45d0-4685-a9cd-bbeb9ab78bea" containerName="revoke-keys"
Nov 03 11:21:42.798234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.798217    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013cd32c-45d0-4685-a9cd-bbeb9ab78bea" containerName="revoke-keys"
Nov 03 11:21:42.798639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.798618    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="013cd32c-45d0-4685-a9cd-bbeb9ab78bea" containerName="revoke-keys"
Nov 03 11:21:42.816902 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.816872    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:42.827647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.823808    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:21:42.855986 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.855796    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"]
Nov 03 11:21:42.910064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.910040    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:21:42.921429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:42.921402    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:21:43.006523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.006489    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-plugins-config-volume\") pod \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\" (UID: \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\") "
Nov 03 11:21:43.006769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.006539    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-kube-api-access-kd4vc\" (UniqueName: \"kubernetes.io/projected/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-kube-api-access-kd4vc\") pod \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\" (UID: \"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e\") "
Nov 03 11:21:43.006943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.006921    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-plugins-config-volume" pod "3d5d9e99-aa64-4a6d-a11d-1617ef5e643e" (UID: "3d5d9e99-aa64-4a6d-a11d-1617ef5e643e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:43.007323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.007264    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.007528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.007403    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stzsv\" (UniqueName: \"kubernetes.io/projected/946cee10-f347-4ead-a471-f613ec4f85e9-kube-api-access-stzsv\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.007528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.007451    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.007805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.007751    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:43.010084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.010061    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-kube-api-access-kd4vc" pod "3d5d9e99-aa64-4a6d-a11d-1617ef5e643e" (UID: "3d5d9e99-aa64-4a6d-a11d-1617ef5e643e"). InnerVolumeSpecName "kube-api-access-kd4vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:43.108877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.108842    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.109028 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.108943    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzsv\" (UniqueName: \"kubernetes.io/projected/946cee10-f347-4ead-a471-f613ec4f85e9-kube-api-access-stzsv\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.109028 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.108974    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.109199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.109180    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd4vc\" (UniqueName: \"kubernetes.io/projected/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e-kube-api-access-kd4vc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:43.110006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.109981    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.111746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.111724    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.117434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.117411    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzsv\" (UniqueName: \"kubernetes.io/projected/946cee10-f347-4ead-a471-f613ec4f85e9-kube-api-access-stzsv\") pod \"maas-api-revoke-sub-14a3e95b-slfjd\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.170792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.170766    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:43.191943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.191742    5817 generic.go:356] "Generic (PLEG): container finished" podID="3d5d9e99-aa64-4a6d-a11d-1617ef5e643e" containerID="e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487" exitCode=0
Nov 03 11:21:43.191943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.191806    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"
Nov 03 11:21:43.191943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.191828    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj" event={"ID":"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e","Type":"ContainerDied","Data":"e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487"}
Nov 03 11:21:43.191943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.191896    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj" event={"ID":"3d5d9e99-aa64-4a6d-a11d-1617ef5e643e","Type":"ContainerDied","Data":"25a71fcb91f6ef9ed38fcd778a551c9c123e8ed1219af9161de3cc9baf10cbbb"}
Nov 03 11:21:43.191943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.191929    5817 scope.go:122] "RemoveContainer" containerID="e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487"
Nov 03 11:21:43.205982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.205962    5817 scope.go:122] "RemoveContainer" containerID="e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487"
Nov 03 11:21:43.206309 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:21:43.206289    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487\": container with ID starting with e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487 not found: ID does not exist" containerID="e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487"
Nov 03 11:21:43.206396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.206316    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487"} err="failed to get container status \"e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487\": rpc error: code = NotFound desc = could not find container \"e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487\": container with ID starting with e1994f82243b244feac0eb0de0f2265e405baf4d97de3147bb9b7a1b2d9e8487 not found: ID does not exist"
Nov 03 11:21:43.225658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.225618    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"]
Nov 03 11:21:43.234895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.234864    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055d82-78f96f8749-b4vpj"]
Nov 03 11:21:43.478665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.478540    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:43.581342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.581314    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"]
Nov 03 11:21:43.584498 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:21:43.584473    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946cee10_f347_4ead_a471_f613ec4f85e9.slice/crio-1b2a25a00d3e3617487b6c680e5f299e99d85388b5f6a797ebbf8d0f7be87a6b WatchSource:0}: Error finding container 1b2a25a00d3e3617487b6c680e5f299e99d85388b5f6a797ebbf8d0f7be87a6b: Status 404 returned error can't find the container with id 1b2a25a00d3e3617487b6c680e5f299e99d85388b5f6a797ebbf8d0f7be87a6b
Nov 03 11:21:43.909723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:43.909686    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5d9e99-aa64-4a6d-a11d-1617ef5e643e" path="/var/lib/kubelet/pods/3d5d9e99-aa64-4a6d-a11d-1617ef5e643e/volumes"
Nov 03 11:21:44.086618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:44.082604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:44.210388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:44.210358    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd" event={"ID":"946cee10-f347-4ead-a471-f613ec4f85e9","Type":"ContainerStarted","Data":"cd79a6b03f01c7990ff94b71d73f17f5907ec563b5d8f4b2519646d9b416450d"}
Nov 03 11:21:44.210719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:44.210419    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd" event={"ID":"946cee10-f347-4ead-a471-f613ec4f85e9","Type":"ContainerStarted","Data":"1b2a25a00d3e3617487b6c680e5f299e99d85388b5f6a797ebbf8d0f7be87a6b"}
Nov 03 11:21:44.226705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:44.226663    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd" podStartSLOduration=2.226650661 podStartE2EDuration="2.226650661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:21:44.225306558 +0000 UTC m=+3134.430888610" watchObservedRunningTime="2026-11-03 11:21:44.226650661 +0000 UTC m=+3134.432232711"
Nov 03 11:21:44.619668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:44.619458    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:44.789861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:44.776204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:44.934650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:44.932048    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:45.297709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:45.297611    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"]
Nov 03 11:21:45.302158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:45.302130    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055d82-gdk8h"]
Nov 03 11:21:45.311986 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:45.311938    5817 generic.go:356] "Generic (PLEG): container finished" podID="946cee10-f347-4ead-a471-f613ec4f85e9" containerID="cd79a6b03f01c7990ff94b71d73f17f5907ec563b5d8f4b2519646d9b416450d" exitCode=0
Nov 03 11:21:45.312557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:45.312528    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd" event={"ID":"946cee10-f347-4ead-a471-f613ec4f85e9","Type":"ContainerDied","Data":"cd79a6b03f01c7990ff94b71d73f17f5907ec563b5d8f4b2519646d9b416450d"}
Nov 03 11:21:45.420601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:45.420027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:45.675032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:45.674952    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:45.907219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:45.907174    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bfa725-687b-4bbc-8686-ff41c579d5d4" path="/var/lib/kubelet/pods/f1bfa725-687b-4bbc-8686-ff41c579d5d4/volumes"
Nov 03 11:21:46.169869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.169714    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:46.553751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.553726    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:46.607349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.607313    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-openshift-ca\") pod \"946cee10-f347-4ead-a471-f613ec4f85e9\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") "
Nov 03 11:21:46.607535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.607511    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946cee10-f347-4ead-a471-f613ec4f85e9-kube-api-access-stzsv\" (UniqueName: \"kubernetes.io/projected/946cee10-f347-4ead-a471-f613ec4f85e9-kube-api-access-stzsv\") pod \"946cee10-f347-4ead-a471-f613ec4f85e9\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") "
Nov 03 11:21:46.607662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.607644    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-xks-ca\") pod \"946cee10-f347-4ead-a471-f613ec4f85e9\" (UID: \"946cee10-f347-4ead-a471-f613ec4f85e9\") "
Nov 03 11:21:46.608294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.608262    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-openshift-ca" pod "946cee10-f347-4ead-a471-f613ec4f85e9" (UID: "946cee10-f347-4ead-a471-f613ec4f85e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:46.608619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.608565    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:46.610426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.610399    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946cee10-f347-4ead-a471-f613ec4f85e9-kube-api-access-stzsv" pod "946cee10-f347-4ead-a471-f613ec4f85e9" (UID: "946cee10-f347-4ead-a471-f613ec4f85e9"). InnerVolumeSpecName "kube-api-access-stzsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:46.610747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.610720    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-xks-ca" pod "946cee10-f347-4ead-a471-f613ec4f85e9" (UID: "946cee10-f347-4ead-a471-f613ec4f85e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:46.710189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.710162    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stzsv\" (UniqueName: \"kubernetes.io/projected/946cee10-f347-4ead-a471-f613ec4f85e9-kube-api-access-stzsv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:46.710566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.710193    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946cee10-f347-4ead-a471-f613ec4f85e9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:46.890653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:46.884387    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:47.337384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:47.337352    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"
Nov 03 11:21:47.337607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:47.337393    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd" event={"ID":"946cee10-f347-4ead-a471-f613ec4f85e9","Type":"ContainerDied","Data":"1b2a25a00d3e3617487b6c680e5f299e99d85388b5f6a797ebbf8d0f7be87a6b"}
Nov 03 11:21:47.337607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:47.337451    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b2a25a00d3e3617487b6c680e5f299e99d85388b5f6a797ebbf8d0f7be87a6b"
Nov 03 11:21:47.537113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:47.537079    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:48.383654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:48.383237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:49.092709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:49.092619    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:51.864990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:51.864957    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:51.996375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:51.996337    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:52.290013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:52.286739    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:52.495598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:52.483404    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:52.653654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:52.645824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:52.801088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:52.801019    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"]
Nov 03 11:21:52.923010 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:21:52.922894    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:21:52.961618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:52.956974    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:53.000653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.000612    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" containerID="cri-o://eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857" gracePeriod=30
Nov 03 11:21:53.309950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.285224    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"]
Nov 03 11:21:53.309950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.285463    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" containerID="cri-o://5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215" gracePeriod=30
Nov 03 11:21:53.360758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.350301    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.696599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.691725    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 11:21:53.696599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.694029    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.729606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.728863    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.843601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.830742    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.882700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.861021    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 11:21:53.882700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.862996    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.882700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.881546    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.953654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:53.937484    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.195676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.184664    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.396651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.383480    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.408881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.408848    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"]
Nov 03 11:21:54.493621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.481880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:54.494400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.494359    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3d5d9e99-aa64-4a6d-a11d-1617ef5e643e" containerName="payload-processing"
Nov 03 11:21:54.494608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.494592    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5d9e99-aa64-4a6d-a11d-1617ef5e643e" containerName="payload-processing"
Nov 03 11:21:54.494967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.494857    5817 cpu_manager.go:414] "containerMap: removing container" podUID="946cee10-f347-4ead-a471-f613ec4f85e9" containerName="revoke-keys"
Nov 03 11:21:54.495115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.495101    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946cee10-f347-4ead-a471-f613ec4f85e9" containerName="revoke-keys"
Nov 03 11:21:54.496145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.496116    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="946cee10-f347-4ead-a471-f613ec4f85e9" containerName="revoke-keys"
Nov 03 11:21:54.501846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.501827    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5d9e99-aa64-4a6d-a11d-1617ef5e643e" containerName="payload-processing"
Nov 03 11:21:54.698716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.698666    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 11:21:54.744632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.702996    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 11:21:54.757646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.749686    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.792601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.767422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:54.838599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.837930    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 11:21:54.838599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.837978    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"]
Nov 03 11:21:54.838599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.838062    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 11:21:54.838599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.838127    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 11:21:54.838599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.838187    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 11:21:54.861596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.839911    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:54.894651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.885862    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 11:21:54.912753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.905691    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:21:54.912753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.906678    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 11:21:54.912753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.907466    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 11:21:54.913947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.913914    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.944028 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.944000    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 11:21:54.986473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.986390    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 11:21:54.987227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.987207    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 11:21:54.992594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.965874    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.225601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.178229    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 11:21:55.321664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:54.976867    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:21:55.389093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.230338    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 11:21:55.416616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.414272    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 11:21:55.580804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.551255    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 11:21:55.580804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.564148    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:55.625608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.596788    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:55.652768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.643265    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:55.652768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.643643    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlc5f\" (UniqueName: \"kubernetes.io/projected/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-kube-api-access-nlc5f\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:55.781592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.762843    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 11:21:55.818850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.804918    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.859241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.775147    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-lrkj6" podUID="618e5bc7-d6bb-4234-80c5-a3da85ed8cb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.943272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.943224    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.946445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.946385    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.947424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.946715    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.996170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:55.996127    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:56.089412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.089369    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 11:21:56.235104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.235057    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 11:21:56.324521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.324285    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Nov 03 11:21:56.434596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.414700    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 11:21:56.569775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.569683    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 11:21:56.579977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.579937    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:56.580545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.580519    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:56.580839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.580819    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlc5f\" (UniqueName: \"kubernetes.io/projected/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-kube-api-access-nlc5f\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:56.644599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.622957    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:56.776598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.768916    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 11:21:57.018105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.797974    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:57.028126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.981561    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlc5f\" (UniqueName: \"kubernetes.io/projected/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-kube-api-access-nlc5f\") pod \"maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:57.028326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:56.991770    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 11:21:57.052906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.009312    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 11:21:57.053370 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:21:57.010615    5817 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.07s"
Nov 03 11:21:57.053554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.053537    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 11:21:57.053731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.053719    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 11:21:57.053866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.053855    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 11:21:57.059635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.009274    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:57.237598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.236750    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:57.274683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.239637    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:57.291677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.291641    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:21:57.291915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.291887    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:21:57.303897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.294531    5817 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:21:57.352977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.352938    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:21:57.353216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.334503    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:21:57.526667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.526325    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:57.658597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.646883    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:57.658597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.653895    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 11:21:57.731993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.719024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:57.931650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:57.928427    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Nov 03 11:21:58.019599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.015938    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded"
Nov 03 11:21:58.046741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.022514    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 11:21:58.083714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.076743    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:58.083714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.080405    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:58.083714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.083346    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 03 11:21:58.236647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.213823    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:58.245590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.239513    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:58.350068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.250160    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:58.350068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.276832    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 11:21:58.579800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.563911    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 11:21:58.579800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.564825    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 03 11:21:58.642296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.642032    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:58.648491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.647652    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:58.803597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.771427    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 11:21:58.910659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.859301    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 11:21:58.910659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.869362    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 11:21:58.910659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:58.869724    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 11:21:59.116791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.083836    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:59.116791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.106728    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw"
Nov 03 11:21:59.184681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.183382    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:21:59.187596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.185959    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 11:21:59.462272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.461945    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 11:21:59.512993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.484208    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 11:21:59.566624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.565193    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:59.655546 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:21:59.655487    5817 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf85ac22_1bd8_4539_8041_a8ff4a4f88a0.slice/crio-3ee2ca1d7661b26c2d70eb38e33c8b056605048c6301f0d6294268aa9ace8b89\": RecentStats: unable to find data in memory cache]"
Nov 03 11:21:59.657851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.657823    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 11:21:59.659030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.658993    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:59.663455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.663422    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2" event={"ID":"df85ac22-1bd8-4539-8041-a8ff4a4f88a0","Type":"ContainerStarted","Data":"3ee2ca1d7661b26c2d70eb38e33c8b056605048c6301f0d6294268aa9ace8b89"}
Nov 03 11:21:59.757693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.757654    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"]
Nov 03 11:21:59.775724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:21:59.775546    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:00.166087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.165749    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:22:00.270906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.270868    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bed6f31-3335-410e-a924-a345d610e597-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bed6f31-3335-410e-a924-a345d610e597-istio-podinfo\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.271099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.270931    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-certs\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.271099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.271024    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-istio-token\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-istio-token\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.271249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.271211    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-envoy\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.271314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.271267    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-credential-socket\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.271376 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.271344    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-kube-api-access-mngfc\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-kube-api-access-mngfc\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.271928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.271435    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-socket\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.271928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.271482    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bed6f31-3335-410e-a924-a345d610e597-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bed6f31-3335-410e-a924-a345d610e597-istiod-ca-cert\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.271928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.271545    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-certs" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.271928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.271587    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-data\") pod \"2bed6f31-3335-410e-a924-a345d610e597\" (UID: \"2bed6f31-3335-410e-a924-a345d610e597\") "
Nov 03 11:22:00.273644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.273596    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.274260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.274239    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-socket" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.274706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.274687    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bed6f31-3335-410e-a924-a345d610e597-istiod-ca-cert" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:00.275071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.275044    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-data" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.275457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.275396    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-credential-socket" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.277446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.277297    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-kube-api-access-mngfc" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "kube-api-access-mngfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:00.277446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.277298    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bed6f31-3335-410e-a924-a345d610e597-istio-podinfo" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:00.277446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.277433    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-envoy" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.281854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.281825    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-istio-token" pod "2bed6f31-3335-410e-a924-a345d610e597" (UID: "2bed6f31-3335-410e-a924-a345d610e597"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:00.375802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.375765    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bed6f31-3335-410e-a924-a345d610e597-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.376051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.376030    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.376179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.376164    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bed6f31-3335-410e-a924-a345d610e597-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.376283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.376270    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.376376 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.376365    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.376470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.376459    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.376590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.376558    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mngfc\" (UniqueName: \"kubernetes.io/projected/2bed6f31-3335-410e-a924-a345d610e597-kube-api-access-mngfc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.376713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.376699    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bed6f31-3335-410e-a924-a345d610e597-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.452386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.452357    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:22:00.581074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581036    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-credential-socket\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.581264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581085    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1743f219-4ae7-463e-956b-97e3ec89561f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743f219-4ae7-463e-956b-97e3ec89561f-istiod-ca-cert\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.581347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581266    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-istio-token\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-istio-token\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.581419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581345    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-credential-socket" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.581419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581386    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-data\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.581543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581460    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-socket\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.581543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581503    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-kube-api-access-5c4xv\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-kube-api-access-5c4xv\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.581543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581535    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-envoy\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.581769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1743f219-4ae7-463e-956b-97e3ec89561f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743f219-4ae7-463e-956b-97e3ec89561f-istio-podinfo\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.581769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581671    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1743f219-4ae7-463e-956b-97e3ec89561f-istiod-ca-cert" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:00.581769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581678    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-data" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.581769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581760    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-socket" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.581992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581972    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-certs" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.582045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.581995    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-certs\") pod \"1743f219-4ae7-463e-956b-97e3ec89561f\" (UID: \"1743f219-4ae7-463e-956b-97e3ec89561f\") "
Nov 03 11:22:00.582936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.582911    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.583049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.582940    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743f219-4ae7-463e-956b-97e3ec89561f-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.583049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.582955    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.583049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.582968    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.583049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.582981    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.584303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.584277    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-kube-api-access-5c4xv" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "kube-api-access-5c4xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:00.584388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.584370    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-istio-token" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:00.584821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.584788    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-envoy" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:00.624055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.623927    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1743f219-4ae7-463e-956b-97e3ec89561f-istio-podinfo" pod "1743f219-4ae7-463e-956b-97e3ec89561f" (UID: "1743f219-4ae7-463e-956b-97e3ec89561f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:00.669611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.666439    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:00.695937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.693761    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c4xv\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-kube-api-access-5c4xv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.695937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.693802    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743f219-4ae7-463e-956b-97e3ec89561f-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.695937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.693819    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743f219-4ae7-463e-956b-97e3ec89561f-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.695937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.693833    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1743f219-4ae7-463e-956b-97e3ec89561f-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.761166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.761058    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2" event={"ID":"df85ac22-1bd8-4539-8041-a8ff4a4f88a0","Type":"ContainerStarted","Data":"b780b71fa03313594a614fd9ee20374635d8d1ff034f293147181f0518959096"}
Nov 03 11:22:00.763675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.763034    5817 generic.go:356] "Generic (PLEG): container finished" podID="2bed6f31-3335-410e-a924-a345d610e597" containerID="5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215" exitCode=0
Nov 03 11:22:00.763675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.763163    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" event={"ID":"2bed6f31-3335-410e-a924-a345d610e597","Type":"ContainerDied","Data":"5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215"}
Nov 03 11:22:00.763675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.763208    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c" event={"ID":"2bed6f31-3335-410e-a924-a345d610e597","Type":"ContainerDied","Data":"0a54b503b8e961e9bf43c280223a7fb8dab371b4e5ee8e101fb5bfecc8050226"}
Nov 03 11:22:00.763675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.763236    5817 scope.go:122] "RemoveContainer" containerID="5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215"
Nov 03 11:22:00.763675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.763405    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"
Nov 03 11:22:00.771750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.769969    5817 generic.go:356] "Generic (PLEG): container finished" podID="1743f219-4ae7-463e-956b-97e3ec89561f" containerID="eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857" exitCode=0
Nov 03 11:22:00.771750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.770132    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" event={"ID":"1743f219-4ae7-463e-956b-97e3ec89561f","Type":"ContainerDied","Data":"eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857"}
Nov 03 11:22:00.771750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.770174    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp" event={"ID":"1743f219-4ae7-463e-956b-97e3ec89561f","Type":"ContainerDied","Data":"a9d1e7662a64fce535b4517f009427226e17385364879b1a2b1b2046208d5de4"}
Nov 03 11:22:00.771750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.770240    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"
Nov 03 11:22:00.786432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.786390    5817 scope.go:122] "RemoveContainer" containerID="5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215"
Nov 03 11:22:00.803067 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:00.786965    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215\": container with ID starting with 5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215 not found: ID does not exist" containerID="5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215"
Nov 03 11:22:00.803067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.787003    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215"} err="failed to get container status \"5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215\": rpc error: code = NotFound desc = could not find container \"5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215\": container with ID starting with 5e5b884a726f5836218d5b118f373c0c3cf8ecbc9bd1a9d886fa92d3c0f1f215 not found: ID does not exist"
Nov 03 11:22:00.803067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.787026    5817 scope.go:122] "RemoveContainer" containerID="eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857"
Nov 03 11:22:00.803067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.790846    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2" podStartSLOduration=7.790829098 podStartE2EDuration="7.790829098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:22:00.780683455 +0000 UTC m=+3150.986265507" watchObservedRunningTime="2026-11-03 11:22:00.790829098 +0000 UTC m=+3150.996411150"
Nov 03 11:22:00.827341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.827305    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"]
Nov 03 11:22:00.838735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.836875    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-b9l6c"]
Nov 03 11:22:00.842611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.842552    5817 scope.go:122] "RemoveContainer" containerID="eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857"
Nov 03 11:22:00.842737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.842718    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"]
Nov 03 11:22:00.843009 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:00.842933    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857\": container with ID starting with eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857 not found: ID does not exist" containerID="eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857"
Nov 03 11:22:00.843009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.842959    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857"} err="failed to get container status \"eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857\": rpc error: code = NotFound desc = could not find container \"eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857\": container with ID starting with eff7caf7d6c04b68ed700cb814778757f8480ac3a1c94fcfd11d8e4971ed3857 not found: ID does not exist"
Nov 03 11:22:00.847995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:00.847966    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-q64dp"]
Nov 03 11:22:01.324691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:01.324632    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"]
Nov 03 11:22:01.325094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:01.324873    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy" containerID="cri-o://8f4aaa1dfa226e832654b437e98f476f8265bff606701159618bf3982058b364" gracePeriod=30
Nov 03 11:22:01.356838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:01.356802    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:01.822485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:01.822455    5817 generic.go:356] "Generic (PLEG): container finished" podID="df85ac22-1bd8-4539-8041-a8ff4a4f88a0" containerID="b780b71fa03313594a614fd9ee20374635d8d1ff034f293147181f0518959096" exitCode=0
Nov 03 11:22:01.822784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:01.822759    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2" event={"ID":"df85ac22-1bd8-4539-8041-a8ff4a4f88a0","Type":"ContainerDied","Data":"b780b71fa03313594a614fd9ee20374635d8d1ff034f293147181f0518959096"}
Nov 03 11:22:01.909047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:01.909016    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" path="/var/lib/kubelet/pods/1743f219-4ae7-463e-956b-97e3ec89561f/volumes"
Nov 03 11:22:01.909710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:01.909694    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bed6f31-3335-410e-a924-a345d610e597" path="/var/lib/kubelet/pods/2bed6f31-3335-410e-a924-a345d610e597/volumes"
Nov 03 11:22:02.076749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:02.076651    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:02.900774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:02.900665    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:03.144182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.144146    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:22:03.304710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.304675    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-xks-ca\") pod \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") "
Nov 03 11:22:03.304880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.304726    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-openshift-ca\") pod \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") "
Nov 03 11:22:03.304962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.304880    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-kube-api-access-nlc5f\" (UniqueName: \"kubernetes.io/projected/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-kube-api-access-nlc5f\") pod \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\" (UID: \"df85ac22-1bd8-4539-8041-a8ff4a4f88a0\") "
Nov 03 11:22:03.305125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.305099    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-openshift-ca" pod "df85ac22-1bd8-4539-8041-a8ff4a4f88a0" (UID: "df85ac22-1bd8-4539-8041-a8ff4a4f88a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:03.305716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.305693    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:03.307192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.307160    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-kube-api-access-nlc5f" pod "df85ac22-1bd8-4539-8041-a8ff4a4f88a0" (UID: "df85ac22-1bd8-4539-8041-a8ff4a4f88a0"). InnerVolumeSpecName "kube-api-access-nlc5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:03.307442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.307414    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-xks-ca" pod "df85ac22-1bd8-4539-8041-a8ff4a4f88a0" (UID: "df85ac22-1bd8-4539-8041-a8ff4a4f88a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:03.407727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.407646    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:03.407727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.407688    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlc5f\" (UniqueName: \"kubernetes.io/projected/df85ac22-1bd8-4539-8041-a8ff4a4f88a0-kube-api-access-nlc5f\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:03.591914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.591881    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:03.853254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.853224    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"
Nov 03 11:22:03.853254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.853204    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2" event={"ID":"df85ac22-1bd8-4539-8041-a8ff4a4f88a0","Type":"ContainerDied","Data":"3ee2ca1d7661b26c2d70eb38e33c8b056605048c6301f0d6294268aa9ace8b89"}
Nov 03 11:22:03.853488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:03.853281    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee2ca1d7661b26c2d70eb38e33c8b056605048c6301f0d6294268aa9ace8b89"
Nov 03 11:22:04.338604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:04.330484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:05.068415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:05.068283    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:05.200594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:05.198885    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"]
Nov 03 11:22:05.315029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:05.314978    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:05.507686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:05.496506    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:06.017048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:06.016971    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:06.397245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:06.397192    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"]
Nov 03 11:22:06.398415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:06.398113    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg" podUID="ab02689b-7c6e-4341-91b4-269a7c51e346" containerName="maas-api" containerID="cri-o://7a04e4e153fc7d2c044ff407c01d8777cf1a9d1ff1a9ebc495dc4db9c0176c1a" gracePeriod=30
Nov 03 11:22:06.482085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:06.481889    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:06.693638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:06.689526    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:07.050495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.050455    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:07.054181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.054151    5817 generic.go:356] "Generic (PLEG): container finished" podID="ab02689b-7c6e-4341-91b4-269a7c51e346" containerID="7a04e4e153fc7d2c044ff407c01d8777cf1a9d1ff1a9ebc495dc4db9c0176c1a" exitCode=0
Nov 03 11:22:07.054509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.054488    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg" event={"ID":"ab02689b-7c6e-4341-91b4-269a7c51e346","Type":"ContainerDied","Data":"7a04e4e153fc7d2c044ff407c01d8777cf1a9d1ff1a9ebc495dc4db9c0176c1a"}
Nov 03 11:22:07.110235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.110200    5817 generic.go:356] "Generic (PLEG): container finished" podID="061e1506-6c9f-4227-8f26-567737295e48" containerID="8f4aaa1dfa226e832654b437e98f476f8265bff606701159618bf3982058b364" exitCode=0
Nov 03 11:22:07.110748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.110724    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" event={"ID":"061e1506-6c9f-4227-8f26-567737295e48","Type":"ContainerDied","Data":"8f4aaa1dfa226e832654b437e98f476f8265bff606701159618bf3982058b364"}
Nov 03 11:22:07.110926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.110913    5817 scope.go:122] "RemoveContainer" containerID="a75baeef46cac0b4aa16d284aa3713cae682bcfef6a615a3ba6d55c6b89a7743"
Nov 03 11:22:07.217317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.216963    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:07.252666 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.250368    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341181    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-data\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341258    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/061e1506-6c9f-4227-8f26-567737295e48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e1506-6c9f-4227-8f26-567737295e48-istio-podinfo\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341395    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-istio-token\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-istio-token\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341441    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061e1506-6c9f-4227-8f26-567737295e48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e1506-6c9f-4227-8f26-567737295e48-istiod-ca-cert\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341471    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-socket\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341550    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-credential-socket\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341652    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-certs\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341802    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-envoy\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.341879    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-kube-api-access-85hjw\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-kube-api-access-85hjw\") pod \"061e1506-6c9f-4227-8f26-567737295e48\" (UID: \"061e1506-6c9f-4227-8f26-567737295e48\") "
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.342668    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061e1506-6c9f-4227-8f26-567737295e48-istiod-ca-cert" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:07.342744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.342518    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-credential-socket" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:07.343375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.342917    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-socket" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:07.343375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.342964    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e1506-6c9f-4227-8f26-567737295e48-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.343479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.343385    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-certs" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:07.343479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.343426    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-data" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:07.347167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.347017    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-istio-token" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:07.347167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.347137    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/061e1506-6c9f-4227-8f26-567737295e48-istio-podinfo" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:07.348735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.348705    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-kube-api-access-85hjw" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "kube-api-access-85hjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:07.348978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.348950    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-envoy" pod "061e1506-6c9f-4227-8f26-567737295e48" (UID: "061e1506-6c9f-4227-8f26-567737295e48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:07.414316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.413918    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:22:07.444047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.444013    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab02689b-7c6e-4341-91b4-269a7c51e346-kube-api-access-nxbdd\" (UniqueName: \"kubernetes.io/projected/ab02689b-7c6e-4341-91b4-269a7c51e346-kube-api-access-nxbdd\") pod \"ab02689b-7c6e-4341-91b4-269a7c51e346\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") "
Nov 03 11:22:07.444434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.444412    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-metrics-certs\") pod \"ab02689b-7c6e-4341-91b4-269a7c51e346\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") "
Nov 03 11:22:07.444551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.444478    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls\") pod \"ab02689b-7c6e-4341-91b4-269a7c51e346\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") "
Nov 03 11:22:07.444551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.444544    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab02689b-7c6e-4341-91b4-269a7c51e346-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab02689b-7c6e-4341-91b4-269a7c51e346-openshift-service-ca\") pod \"ab02689b-7c6e-4341-91b4-269a7c51e346\" (UID: \"ab02689b-7c6e-4341-91b4-269a7c51e346\") "
Nov 03 11:22:07.445289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445139    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab02689b-7c6e-4341-91b4-269a7c51e346-openshift-service-ca" pod "ab02689b-7c6e-4341-91b4-269a7c51e346" (UID: "ab02689b-7c6e-4341-91b4-269a7c51e346"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:07.445783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445755    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.445912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445786    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.445912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445802    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.445912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445816    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.445912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445828    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hjw\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-kube-api-access-85hjw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.445912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445843    5817 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab02689b-7c6e-4341-91b4-269a7c51e346-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.445912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445856    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e1506-6c9f-4227-8f26-567737295e48-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.445912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445871    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e1506-6c9f-4227-8f26-567737295e48-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.445912 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.445883    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061e1506-6c9f-4227-8f26-567737295e48-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.447324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.447294    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls" pod "ab02689b-7c6e-4341-91b4-269a7c51e346" (UID: "ab02689b-7c6e-4341-91b4-269a7c51e346"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:07.448010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.447981    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-metrics-certs" pod "ab02689b-7c6e-4341-91b4-269a7c51e346" (UID: "ab02689b-7c6e-4341-91b4-269a7c51e346"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:07.449317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.449288    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab02689b-7c6e-4341-91b4-269a7c51e346-kube-api-access-nxbdd" pod "ab02689b-7c6e-4341-91b4-269a7c51e346" (UID: "ab02689b-7c6e-4341-91b4-269a7c51e346"). InnerVolumeSpecName "kube-api-access-nxbdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:07.570679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.570642    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:07.571517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.571487    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxbdd\" (UniqueName: \"kubernetes.io/projected/ab02689b-7c6e-4341-91b4-269a7c51e346-kube-api-access-nxbdd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.571669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.571526    5817 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.571669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:07.571545    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab02689b-7c6e-4341-91b4-269a7c51e346-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:08.004600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.004043    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:08.051424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.051389    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"]
Nov 03 11:22:08.091390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.051641    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy" containerID="cri-o://a8d5e75f2f0c84767ff023d57db073a7abd3cf9b1ff89b9989b11dc7a402ccf3" gracePeriod=30
Nov 03 11:22:08.145841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.145801    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg" event={"ID":"ab02689b-7c6e-4341-91b4-269a7c51e346","Type":"ContainerDied","Data":"07213194218b94a473a318015f47b2952c95daa94c3a8e2ef89599d54b15b222"}
Nov 03 11:22:08.146060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.145886    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"
Nov 03 11:22:08.146060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.145915    5817 scope.go:122] "RemoveContainer" containerID="7a04e4e153fc7d2c044ff407c01d8777cf1a9d1ff1a9ebc495dc4db9c0176c1a"
Nov 03 11:22:08.150182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.150151    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx" event={"ID":"061e1506-6c9f-4227-8f26-567737295e48","Type":"ContainerDied","Data":"1072f94b7715b6ffe848cd2d860ee1e9685417a05967e6321a78ca5ab0c70cf8"}
Nov 03 11:22:08.150311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.150249    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"
Nov 03 11:22:08.161382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.161359    5817 scope.go:122] "RemoveContainer" containerID="8f4aaa1dfa226e832654b437e98f476f8265bff606701159618bf3982058b364"
Nov 03 11:22:08.190542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.190504    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"]
Nov 03 11:22:08.196488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.196461    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a68103-65fd7655cc-bkdrg"]
Nov 03 11:22:08.206828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.206801    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"]
Nov 03 11:22:08.212873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.212851    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-bmzgx"]
Nov 03 11:22:08.406722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.406635    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:08.751629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:08.750653    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:09.162772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.162693    5817 generic.go:356] "Generic (PLEG): container finished" podID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerID="204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec" exitCode=0
Nov 03 11:22:09.162772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.162761    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" event={"ID":"8a5308b6-3a71-4bca-9b07-fe824d145610","Type":"ContainerDied","Data":"204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec"}
Nov 03 11:22:09.163222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.163207    5817 scope.go:122] "RemoveContainer" containerID="204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec"
Nov 03 11:22:09.165338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.165309    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:09.165445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.165394    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:09.165524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.165441    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:09.400273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.400142    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:09.909460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.909431    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061e1506-6c9f-4227-8f26-567737295e48" path="/var/lib/kubelet/pods/061e1506-6c9f-4227-8f26-567737295e48/volumes"
Nov 03 11:22:09.909993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.909977    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab02689b-7c6e-4341-91b4-269a7c51e346" path="/var/lib/kubelet/pods/ab02689b-7c6e-4341-91b4-269a7c51e346/volumes"
Nov 03 11:22:09.993597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:09.981732    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:10.177512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.177426    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" event={"ID":"8a5308b6-3a71-4bca-9b07-fe824d145610","Type":"ContainerStarted","Data":"3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191"}
Nov 03 11:22:10.201701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.201652    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs" podUID="c6c3035c-e422-45aa-8a7e-100b3e46d1ab" containerName="payload-processing" containerID="cri-o://dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10" gracePeriod=25
Nov 03 11:22:10.627905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.627870    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:22:10.659000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.658968    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:22:10.660091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.660067    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 11:22:10.660212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.660120    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 11:22:10.754566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.754480    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-plugins-config-volume\") pod \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\" (UID: \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\") "
Nov 03 11:22:10.755336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.755305    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-kube-api-access-dvfrc\" (UniqueName: \"kubernetes.io/projected/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-kube-api-access-dvfrc\") pod \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\" (UID: \"c6c3035c-e422-45aa-8a7e-100b3e46d1ab\") "
Nov 03 11:22:10.764163 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.759365    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:10.764771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.764742    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-plugins-config-volume" pod "c6c3035c-e422-45aa-8a7e-100b3e46d1ab" (UID: "c6c3035c-e422-45aa-8a7e-100b3e46d1ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:10.808104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.808068    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-kube-api-access-dvfrc" pod "c6c3035c-e422-45aa-8a7e-100b3e46d1ab" (UID: "c6c3035c-e422-45aa-8a7e-100b3e46d1ab"). InnerVolumeSpecName "kube-api-access-dvfrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:10.858735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.858703    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvfrc\" (UniqueName: \"kubernetes.io/projected/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-kube-api-access-dvfrc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.858735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:10.858731    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c3035c-e422-45aa-8a7e-100b3e46d1ab-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:11.219331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.217901    5817 generic.go:356] "Generic (PLEG): container finished" podID="c6c3035c-e422-45aa-8a7e-100b3e46d1ab" containerID="dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10" exitCode=0
Nov 03 11:22:11.219331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.218005    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"
Nov 03 11:22:11.219331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.218650    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs" event={"ID":"c6c3035c-e422-45aa-8a7e-100b3e46d1ab","Type":"ContainerDied","Data":"dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10"}
Nov 03 11:22:11.219331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.219134    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs" event={"ID":"c6c3035c-e422-45aa-8a7e-100b3e46d1ab","Type":"ContainerDied","Data":"24be6628127109abc0b703f4670cf7d5b8a91a7b954e25fbea489645a93564a0"}
Nov 03 11:22:11.219331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.219170    5817 scope.go:122] "RemoveContainer" containerID="dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10"
Nov 03 11:22:11.235060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.235043    5817 scope.go:122] "RemoveContainer" containerID="dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10"
Nov 03 11:22:11.235404 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:11.235378    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10\": container with ID starting with dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10 not found: ID does not exist" containerID="dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10"
Nov 03 11:22:11.235493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.235415    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10"} err="failed to get container status \"dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10\": rpc error: code = NotFound desc = could not find container \"dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10\": container with ID starting with dbd96738734bbae33f84b505eea37e81ac4eaeccf789ca42bcc3177425c03b10 not found: ID does not exist"
Nov 03 11:22:11.248130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.248105    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"]
Nov 03 11:22:11.253202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.253183    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a68103-7f656d6f67-clkbs"]
Nov 03 11:22:11.426595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.422559    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:11.659294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.659203    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 11:22:11.659294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.659267    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 11:22:11.927429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:11.927351    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c3035c-e422-45aa-8a7e-100b3e46d1ab" path="/var/lib/kubelet/pods/c6c3035c-e422-45aa-8a7e-100b3e46d1ab/volumes"
Nov 03 11:22:12.262599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:12.252682    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:12.658926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:12.658848    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 11:22:12.658926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:12.658905    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 11:22:13.037070 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.037029    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:13.248794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.248764    5817 generic.go:356] "Generic (PLEG): container finished" podID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerID="a8d5e75f2f0c84767ff023d57db073a7abd3cf9b1ff89b9989b11dc7a402ccf3" exitCode=0
Nov 03 11:22:13.248969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.248839    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" event={"ID":"ce91a647-dc88-49fe-929b-0cccd1b19fa0","Type":"ContainerDied","Data":"a8d5e75f2f0c84767ff023d57db073a7abd3cf9b1ff89b9989b11dc7a402ccf3"}
Nov 03 11:22:13.445456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.445429    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:22:13.543997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.543950    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-credential-socket\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544028    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-certs\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544063    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-kube-api-access-w7dtr\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-kube-api-access-w7dtr\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544090    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istiod-ca-cert\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544184    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-podinfo\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544220    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-token\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-token\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544286    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-envoy\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544332    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-socket\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544359    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-data\") pod \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\" (UID: \"ce91a647-dc88-49fe-929b-0cccd1b19fa0\") "
Nov 03 11:22:13.544902 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.544874    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istiod-ca-cert" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:13.545150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.545130    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-credential-socket" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.545414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.545392    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-certs" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.545495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.545451    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-socket" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.545738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.545660    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-data" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.547616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.547551    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-token" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:13.547802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.547741    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-podinfo" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:13.548014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.547993    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-envoy" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.549276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.549231    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:13.553367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.553338    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-kube-api-access-w7dtr" pod "ce91a647-dc88-49fe-929b-0cccd1b19fa0" (UID: "ce91a647-dc88-49fe-929b-0cccd1b19fa0"). InnerVolumeSpecName "kube-api-access-w7dtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666297    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666335    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666349    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666364    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666377    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666392    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666406    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce91a647-dc88-49fe-929b-0cccd1b19fa0-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666420    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7dtr\" (UniqueName: \"kubernetes.io/projected/ce91a647-dc88-49fe-929b-0cccd1b19fa0-kube-api-access-w7dtr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.670708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.666435    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce91a647-dc88-49fe-929b-0cccd1b19fa0-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:13.695594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.690690    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 11:22:13.695594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.690755    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 11:22:13.955738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.955697    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"]
Nov 03 11:22:13.955947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:13.955773    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"]
Nov 03 11:22:14.068941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.068899    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:14.219987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.219864    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:14.341476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.341436    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr" event={"ID":"ce91a647-dc88-49fe-929b-0cccd1b19fa0","Type":"ContainerDied","Data":"b28ff69f70f83e08a599a4c7f84847e8e2b35ea2f0c85d489ebaa825c824098c"}
Nov 03 11:22:14.342265 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.342243    5817 scope.go:122] "RemoveContainer" containerID="a8d5e75f2f0c84767ff023d57db073a7abd3cf9b1ff89b9989b11dc7a402ccf3"
Nov 03 11:22:14.342559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.342147    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"
Nov 03 11:22:14.468601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.466989    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:14.595709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.548439    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"]
Nov 03 11:22:14.595709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.548739    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" containerID="cri-o://d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835" gracePeriod=30
Nov 03 11:22:14.674600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.670613    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 11:22:14.674600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.670673    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 11:22:14.776608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.773020    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"]
Nov 03 11:22:14.922872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:14.920772    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:15.122251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.073399    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-l7nwr"]
Nov 03 11:22:15.243178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.243141    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"]
Nov 03 11:22:15.245813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.245788    5817 cpu_manager.go:414] "containerMap: removing container" podUID="df85ac22-1bd8-4539-8041-a8ff4a4f88a0" containerName="revoke-keys"
Nov 03 11:22:15.245987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.245972    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df85ac22-1bd8-4539-8041-a8ff4a4f88a0" containerName="revoke-keys"
Nov 03 11:22:15.246132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246122    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy"
Nov 03 11:22:15.246214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246204    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy"
Nov 03 11:22:15.246309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246300    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ab02689b-7c6e-4341-91b4-269a7c51e346" containerName="maas-api"
Nov 03 11:22:15.246391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246381    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab02689b-7c6e-4341-91b4-269a7c51e346" containerName="maas-api"
Nov 03 11:22:15.246478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246467    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c6c3035c-e422-45aa-8a7e-100b3e46d1ab" containerName="payload-processing"
Nov 03 11:22:15.246555 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246545    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c3035c-e422-45aa-8a7e-100b3e46d1ab" containerName="payload-processing"
Nov 03 11:22:15.246693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246682    5817 cpu_manager.go:414] "containerMap: removing container" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy"
Nov 03 11:22:15.246780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246770    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy"
Nov 03 11:22:15.246872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246863    5817 cpu_manager.go:414] "containerMap: removing container" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy"
Nov 03 11:22:15.246950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.246940    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy"
Nov 03 11:22:15.247027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247018    5817 cpu_manager.go:414] "containerMap: removing container" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy"
Nov 03 11:22:15.247106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247097    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy"
Nov 03 11:22:15.247194 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247185    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy"
Nov 03 11:22:15.247271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247262    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy"
Nov 03 11:22:15.247534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247523    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy"
Nov 03 11:22:15.247643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247633    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="061e1506-6c9f-4227-8f26-567737295e48" containerName="istio-proxy"
Nov 03 11:22:15.247737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247728    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="1743f219-4ae7-463e-956b-97e3ec89561f" containerName="istio-proxy"
Nov 03 11:22:15.247823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247815    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bed6f31-3335-410e-a924-a345d610e597" containerName="istio-proxy"
Nov 03 11:22:15.247897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247889    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="df85ac22-1bd8-4539-8041-a8ff4a4f88a0" containerName="revoke-keys"
Nov 03 11:22:15.247996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.247985    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" containerName="istio-proxy"
Nov 03 11:22:15.248085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.248076    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab02689b-7c6e-4341-91b4-269a7c51e346" containerName="maas-api"
Nov 03 11:22:15.248171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.248162    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c3035c-e422-45aa-8a7e-100b3e46d1ab" containerName="payload-processing"
Nov 03 11:22:15.279529 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.279507    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"]
Nov 03 11:22:15.279673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.279550    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:22:15.279733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.279704    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.283654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.283634    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:22:15.283825 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.283803    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:22:15.455062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.455018    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.455247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.455084    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.455352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.455333    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shkht\" (UniqueName: \"kubernetes.io/projected/f4f34536-d074-41af-9c44-187d5e75f938-kube-api-access-shkht\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.557281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.557183    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shkht\" (UniqueName: \"kubernetes.io/projected/f4f34536-d074-41af-9c44-187d5e75f938-kube-api-access-shkht\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.557773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.557505    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.557773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.557598    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.558309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.558273    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.560326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.560299    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.567000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.566976    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shkht\" (UniqueName: \"kubernetes.io/projected/f4f34536-d074-41af-9c44-187d5e75f938-kube-api-access-shkht\") pod \"maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.618011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.617982    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:15.660172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.660140    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 11:22:15.660291 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.660194    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 11:22:15.934700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.934180    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce91a647-dc88-49fe-929b-0cccd1b19fa0" path="/var/lib/kubelet/pods/ce91a647-dc88-49fe-929b-0cccd1b19fa0/volumes"
Nov 03 11:22:15.939287 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:22:15.939262    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f34536_d074_41af_9c44_187d5e75f938.slice/crio-4bf6d20e74195a072b45a8d5acacfb96682e2de08662a1befd6346c4b5239e38 WatchSource:0}: Error finding container 4bf6d20e74195a072b45a8d5acacfb96682e2de08662a1befd6346c4b5239e38: Status 404 returned error can't find the container with id 4bf6d20e74195a072b45a8d5acacfb96682e2de08662a1befd6346c4b5239e38
Nov 03 11:22:15.940241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:15.940210    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"]
Nov 03 11:22:16.332587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:16.332483    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:16.421602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:16.421536    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28" event={"ID":"f4f34536-d074-41af-9c44-187d5e75f938","Type":"ContainerStarted","Data":"4bf6d20e74195a072b45a8d5acacfb96682e2de08662a1befd6346c4b5239e38"}
Nov 03 11:22:16.659369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:16.659295    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 11:22:16.659369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:16.659346    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 11:22:17.218149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:17.213261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:17.436866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:17.436836    5817 generic.go:356] "Generic (PLEG): container finished" podID="f4f34536-d074-41af-9c44-187d5e75f938" containerID="7f9b56d597bc4789b9cc8d3aa60ad919244f22684025be6cc179ef1c2dcba3a6" exitCode=0
Nov 03 11:22:17.437036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:17.436917    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28" event={"ID":"f4f34536-d074-41af-9c44-187d5e75f938","Type":"ContainerDied","Data":"7f9b56d597bc4789b9cc8d3aa60ad919244f22684025be6cc179ef1c2dcba3a6"}
Nov 03 11:22:17.637535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:17.637503    5817 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:22:17.637777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:17.637555    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:22:17.663778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:17.663754    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:22:17.664826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:17.664800    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:22:18.114596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.095557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:18.630948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.630918    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:18.650449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.650412    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-openshift-ca\") pod \"f4f34536-d074-41af-9c44-187d5e75f938\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") "
Nov 03 11:22:18.650694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.650673    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-xks-ca\") pod \"f4f34536-d074-41af-9c44-187d5e75f938\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") "
Nov 03 11:22:18.650782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.650756    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f34536-d074-41af-9c44-187d5e75f938-kube-api-access-shkht\" (UniqueName: \"kubernetes.io/projected/f4f34536-d074-41af-9c44-187d5e75f938-kube-api-access-shkht\") pod \"f4f34536-d074-41af-9c44-187d5e75f938\" (UID: \"f4f34536-d074-41af-9c44-187d5e75f938\") "
Nov 03 11:22:18.650841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.650819    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-openshift-ca" pod "f4f34536-d074-41af-9c44-187d5e75f938" (UID: "f4f34536-d074-41af-9c44-187d5e75f938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:18.651800 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.651774    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:18.653248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.653222    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-xks-ca" pod "f4f34536-d074-41af-9c44-187d5e75f938" (UID: "f4f34536-d074-41af-9c44-187d5e75f938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:18.653336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.653288    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f34536-d074-41af-9c44-187d5e75f938-kube-api-access-shkht" pod "f4f34536-d074-41af-9c44-187d5e75f938" (UID: "f4f34536-d074-41af-9c44-187d5e75f938"). InnerVolumeSpecName "kube-api-access-shkht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:18.715032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.714956    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:18.753622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.753593    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f34536-d074-41af-9c44-187d5e75f938-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:18.753622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.753620    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shkht\" (UniqueName: \"kubernetes.io/projected/f4f34536-d074-41af-9c44-187d5e75f938-kube-api-access-shkht\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:18.949410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.943127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:18.956301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.956237    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk" podUID="b7f3810a-c5ff-41bf-bd62-8767c026017b" containerName="payload-pre-processing" containerID="cri-o://a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24" gracePeriod=25
Nov 03 11:22:18.961509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:18.961452    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d" podUID="544fda18-938c-4239-82dc-9920f64e3938" containerName="payload-processing" containerID="cri-o://d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda" gracePeriod=25
Nov 03 11:22:19.236058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.235996    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:22:19.264418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.264386    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544fda18-938c-4239-82dc-9920f64e3938-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fda18-938c-4239-82dc-9920f64e3938-plugins-config-volume\") pod \"544fda18-938c-4239-82dc-9920f64e3938\" (UID: \"544fda18-938c-4239-82dc-9920f64e3938\") "
Nov 03 11:22:19.264662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.264646    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544fda18-938c-4239-82dc-9920f64e3938-kube-api-access-pr6pm\" (UniqueName: \"kubernetes.io/projected/544fda18-938c-4239-82dc-9920f64e3938-kube-api-access-pr6pm\") pod \"544fda18-938c-4239-82dc-9920f64e3938\" (UID: \"544fda18-938c-4239-82dc-9920f64e3938\") "
Nov 03 11:22:19.264869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.264843    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544fda18-938c-4239-82dc-9920f64e3938-plugins-config-volume" pod "544fda18-938c-4239-82dc-9920f64e3938" (UID: "544fda18-938c-4239-82dc-9920f64e3938"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:19.265322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.265304    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fda18-938c-4239-82dc-9920f64e3938-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:19.267610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.267562    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544fda18-938c-4239-82dc-9920f64e3938-kube-api-access-pr6pm" pod "544fda18-938c-4239-82dc-9920f64e3938" (UID: "544fda18-938c-4239-82dc-9920f64e3938"). InnerVolumeSpecName "kube-api-access-pr6pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:19.367199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.367167    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr6pm\" (UniqueName: \"kubernetes.io/projected/544fda18-938c-4239-82dc-9920f64e3938-kube-api-access-pr6pm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:19.457872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.457844    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"
Nov 03 11:22:19.457872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.457861    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28" event={"ID":"f4f34536-d074-41af-9c44-187d5e75f938","Type":"ContainerDied","Data":"4bf6d20e74195a072b45a8d5acacfb96682e2de08662a1befd6346c4b5239e38"}
Nov 03 11:22:19.458120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.457908    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bf6d20e74195a072b45a8d5acacfb96682e2de08662a1befd6346c4b5239e38"
Nov 03 11:22:19.459514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.459488    5817 generic.go:356] "Generic (PLEG): container finished" podID="544fda18-938c-4239-82dc-9920f64e3938" containerID="d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda" exitCode=0
Nov 03 11:22:19.459689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.459534    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"
Nov 03 11:22:19.459689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.459560    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d" event={"ID":"544fda18-938c-4239-82dc-9920f64e3938","Type":"ContainerDied","Data":"d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda"}
Nov 03 11:22:19.459689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.459629    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d" event={"ID":"544fda18-938c-4239-82dc-9920f64e3938","Type":"ContainerDied","Data":"db5d6e52a55d7dc556afedb72eeb9e3841cd256b37d9655da492e1ed72bef17e"}
Nov 03 11:22:19.459689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.459656    5817 scope.go:122] "RemoveContainer" containerID="d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda"
Nov 03 11:22:19.477388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.477313    5817 scope.go:122] "RemoveContainer" containerID="d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda"
Nov 03 11:22:19.477992 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:19.477968    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda\": container with ID starting with d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda not found: ID does not exist" containerID="d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda"
Nov 03 11:22:19.478092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.478002    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda"} err="failed to get container status \"d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda\": rpc error: code = NotFound desc = could not find container \"d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda\": container with ID starting with d89c7b295a15c8098a424720e0c8a5380305ae6f72cff9bd58de1494488ffeda not found: ID does not exist"
Nov 03 11:22:19.493148 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.492958    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"]
Nov 03 11:22:19.499613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.499590    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24376f-57858f46c-ghv5d"]
Nov 03 11:22:19.804646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.802100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:19.907245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:19.907209    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544fda18-938c-4239-82dc-9920f64e3938" path="/var/lib/kubelet/pods/544fda18-938c-4239-82dc-9920f64e3938/volumes"
Nov 03 11:22:20.390723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.390695    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:22:20.476701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.476606    5817 generic.go:356] "Generic (PLEG): container finished" podID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerID="d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835" exitCode=0
Nov 03 11:22:20.476907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.476696    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"
Nov 03 11:22:20.476907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.476751    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" event={"ID":"a9fce276-e6fe-4835-b146-c9a4642598e5","Type":"ContainerDied","Data":"d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835"}
Nov 03 11:22:20.476907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.476803    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8" event={"ID":"a9fce276-e6fe-4835-b146-c9a4642598e5","Type":"ContainerDied","Data":"947ac77e9f3e76ea2779d62843ee07dc7ccd7bbfe682af19a6a37e5532e46189"}
Nov 03 11:22:20.476907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.476831    5817 scope.go:122] "RemoveContainer" containerID="d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835"
Nov 03 11:22:20.494150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.494125    5817 scope.go:122] "RemoveContainer" containerID="a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867"
Nov 03 11:22:20.505884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.505830    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-envoy\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.505975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.505896    5817 scope.go:122] "RemoveContainer" containerID="d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835"
Nov 03 11:22:20.505975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.505905    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-socket\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.505975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.505944    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-certs\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.505975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.505969    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fce276-e6fe-4835-b146-c9a4642598e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9fce276-e6fe-4835-b146-c9a4642598e5-istiod-ca-cert\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.506204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.506009    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-podinfo\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.506204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.506043    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-credential-socket\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.506204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.506098    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-token\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-token\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.506204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.506118    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-kube-api-access-47t9x\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-kube-api-access-47t9x\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.506204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.506156    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-data\") pod \"a9fce276-e6fe-4835-b146-c9a4642598e5\" (UID: \"a9fce276-e6fe-4835-b146-c9a4642598e5\") "
Nov 03 11:22:20.506204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.506188    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-socket" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.506727 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:20.506702    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835\": container with ID starting with d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835 not found: ID does not exist" containerID="d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835"
Nov 03 11:22:20.506818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.506737    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835"} err="failed to get container status \"d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835\": rpc error: code = NotFound desc = could not find container \"d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835\": container with ID starting with d7d44416c650d9709c43404c58efa3b8e3516cfb2e79ce1cbb718610ebf94835 not found: ID does not exist"
Nov 03 11:22:20.506818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.506760    5817 scope.go:122] "RemoveContainer" containerID="a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867"
Nov 03 11:22:20.507131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.507109    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.507266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.507239    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-credential-socket" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.507373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.507270    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-data" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.507373 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:20.507332    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867\": container with ID starting with a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867 not found: ID does not exist" containerID="a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867"
Nov 03 11:22:20.507515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.507364    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867"} err="failed to get container status \"a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867\": rpc error: code = NotFound desc = could not find container \"a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867\": container with ID starting with a2ea1e279397e3087a0297a59a677c19b75e51f016578c4cdbb0c86e30806867 not found: ID does not exist"
Nov 03 11:22:20.507857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.507832    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fce276-e6fe-4835-b146-c9a4642598e5-istiod-ca-cert" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:20.507984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.507924    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-certs" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.509361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.509326    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-envoy" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.509462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.509418    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-podinfo" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:20.510033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.509999    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-kube-api-access-47t9x" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "kube-api-access-47t9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:20.510116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.510056    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-token" pod "a9fce276-e6fe-4835-b146-c9a4642598e5" (UID: "a9fce276-e6fe-4835-b146-c9a4642598e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:20.601712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.601524    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:20.630440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.615363    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.630440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.615397    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9fce276-e6fe-4835-b146-c9a4642598e5-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.630440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.615413    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.630440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.615428    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.630440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.615442    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.630440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.615456    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47t9x\" (UniqueName: \"kubernetes.io/projected/a9fce276-e6fe-4835-b146-c9a4642598e5-kube-api-access-47t9x\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.630440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.615471    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.630440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.615484    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9fce276-e6fe-4835-b146-c9a4642598e5-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.810318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.810279    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"]
Nov 03 11:22:20.829513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:20.829483    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-8jgl8"]
Nov 03 11:22:21.122141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:21.122038    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:21.566775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:21.566740    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:21.622444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:21.622111    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk" podUID="b7f3810a-c5ff-41bf-bd62-8767c026017b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:22:21.736203 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:21.736163    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:21.736554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:21.736535    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"]
Nov 03 11:22:21.740851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:21.740823    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24376f-mfgh2"]
Nov 03 11:22:21.920404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:21.920305    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" path="/var/lib/kubelet/pods/a9fce276-e6fe-4835-b146-c9a4642598e5/volumes"
Nov 03 11:22:21.921124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:21.921099    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df85ac22-1bd8-4539-8041-a8ff4a4f88a0" path="/var/lib/kubelet/pods/df85ac22-1bd8-4539-8041-a8ff4a4f88a0/volumes"
Nov 03 11:22:22.077630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:22.075451    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:22.256286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:22.256255    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:22.420692 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:22.420657    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:22:22.657768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:22.655105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:23.281599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:23.279114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:24.121594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:24.117958    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:24.405838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:24.405764    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:22:24.683597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:24.682629    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:25.312219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:25.312181    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:25.479534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:25.479497    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:26.452469 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:26.452422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:27.040649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:27.040522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:27.816225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:27.816190    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:28.458492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:28.452317    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:28.758431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:28.758342    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:28.894049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:28.894003    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:29.065072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:29.063742    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"]
Nov 03 11:22:29.065072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:29.063982    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy" containerID="cri-o://ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f" gracePeriod=30
Nov 03 11:22:29.079646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:29.076832    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:29.473543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:29.473511    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:29.867505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:29.867415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:30.171697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:30.171620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:30.307948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:30.307913    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:30.709621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:30.708467    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:31.313599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:31.302584    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:31.611665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:31.611563    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk" podUID="b7f3810a-c5ff-41bf-bd62-8767c026017b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:22:32.180342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:32.180303    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:32.892032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:32.891994    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:33.159866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.159785    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"]
Nov 03 11:22:33.160751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160729    5817 cpu_manager.go:414] "containerMap: removing container" podUID="544fda18-938c-4239-82dc-9920f64e3938" containerName="payload-processing"
Nov 03 11:22:33.160751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160750    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544fda18-938c-4239-82dc-9920f64e3938" containerName="payload-processing"
Nov 03 11:22:33.160943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160771    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy"
Nov 03 11:22:33.160943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160776    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy"
Nov 03 11:22:33.160943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160803    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy"
Nov 03 11:22:33.160943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160810    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy"
Nov 03 11:22:33.160943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160834    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f4f34536-d074-41af-9c44-187d5e75f938" containerName="revoke-keys"
Nov 03 11:22:33.160943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160840    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f34536-d074-41af-9c44-187d5e75f938" containerName="revoke-keys"
Nov 03 11:22:33.161140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160947    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy"
Nov 03 11:22:33.161140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160958    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f34536-d074-41af-9c44-187d5e75f938" containerName="revoke-keys"
Nov 03 11:22:33.161140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160966    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="544fda18-938c-4239-82dc-9920f64e3938" containerName="payload-processing"
Nov 03 11:22:33.161140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.160976    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fce276-e6fe-4835-b146-c9a4642598e5" containerName="istio-proxy"
Nov 03 11:22:33.164918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.164892    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.168690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.168664    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:22:33.169827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.169790    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:22:33.171320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.171286    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"]
Nov 03 11:22:33.266345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.266307    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.266764 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.266465    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.266764 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.266524    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl464\" (UniqueName: \"kubernetes.io/projected/aa059d4c-5a2f-461d-a414-3ca128998417-kube-api-access-fl464\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.368496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.368460    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.368692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.368584    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.368692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.368642    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl464\" (UniqueName: \"kubernetes.io/projected/aa059d4c-5a2f-461d-a414-3ca128998417-kube-api-access-fl464\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.369294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.369273    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.371198 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.371176    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.376330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.376301    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl464\" (UniqueName: \"kubernetes.io/projected/aa059d4c-5a2f-461d-a414-3ca128998417-kube-api-access-fl464\") pod \"maas-api-revoke-sub-ce03bd71-v2vcr\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.484883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.484848    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:33.652482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.652406    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:33.774774 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:22:33.774744    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa059d4c_5a2f_461d_a414_3ca128998417.slice/crio-009202a6a59f0a7ed915b708b57dca27c1de910ef41af49cc1dbf83ae9c65ea5 WatchSource:0}: Error finding container 009202a6a59f0a7ed915b708b57dca27c1de910ef41af49cc1dbf83ae9c65ea5: Status 404 returned error can't find the container with id 009202a6a59f0a7ed915b708b57dca27c1de910ef41af49cc1dbf83ae9c65ea5
Nov 03 11:22:33.778260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.778229    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"]
Nov 03 11:22:33.802634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:33.802600    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr" event={"ID":"aa059d4c-5a2f-461d-a414-3ca128998417","Type":"ContainerStarted","Data":"009202a6a59f0a7ed915b708b57dca27c1de910ef41af49cc1dbf83ae9c65ea5"}
Nov 03 11:22:34.307891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.293843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:34.517292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.517259    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:22:34.647527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647446    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-credential-socket\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.647758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647644    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-podinfo\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.647758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647730    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-socket\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.647871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647770    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-certs\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.647871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647794    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-credential-socket" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.648005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647882    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istiod-ca-cert\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.648005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647902    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-token\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-token\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.648005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647940    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-data\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.648005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647977    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-envoy\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.648005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.647984    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-socket" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.648266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648086    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-kube-api-access-bnpwl\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-kube-api-access-bnpwl\") pod \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\" (UID: \"aa54aa54-6c59-4169-a0f2-beb2ca32557b\") "
Nov 03 11:22:34.648266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648126    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-certs" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.648266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648191    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-data" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.648453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648266    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istiod-ca-cert" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:34.648970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648878    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.648970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648902    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.648970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648915    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.648970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648926    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.648970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.648934    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.650735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.650713    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-kube-api-access-bnpwl" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "kube-api-access-bnpwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:34.650838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.650742    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-token" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:34.650988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.650963    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-envoy" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.650988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.650975    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-podinfo" pod "aa54aa54-6c59-4169-a0f2-beb2ca32557b" (UID: "aa54aa54-6c59-4169-a0f2-beb2ca32557b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:34.750627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.750597    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.750627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.750626    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.750868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.750638    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnpwl\" (UniqueName: \"kubernetes.io/projected/aa54aa54-6c59-4169-a0f2-beb2ca32557b-kube-api-access-bnpwl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.750868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.750650    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa54aa54-6c59-4169-a0f2-beb2ca32557b-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.823639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.819986    5817 generic.go:356] "Generic (PLEG): container finished" podID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerID="ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f" exitCode=0
Nov 03 11:22:34.823639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.820111    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" event={"ID":"aa54aa54-6c59-4169-a0f2-beb2ca32557b","Type":"ContainerDied","Data":"ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f"}
Nov 03 11:22:34.823639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.820160    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq" event={"ID":"aa54aa54-6c59-4169-a0f2-beb2ca32557b","Type":"ContainerDied","Data":"f40126d39a3e05cd1a3f20daa0753626a04ed46af3ebeb3cde23899db6e254e3"}
Nov 03 11:22:34.823639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.820189    5817 scope.go:122] "RemoveContainer" containerID="ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f"
Nov 03 11:22:34.823639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.820351    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"
Nov 03 11:22:34.824841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.824804    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr" event={"ID":"aa059d4c-5a2f-461d-a414-3ca128998417","Type":"ContainerStarted","Data":"e4ab3554cec4f2678797142b16e3873af8a60a05df2dd7ada8729796ac859ab8"}
Nov 03 11:22:34.836776 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.836736    5817 scope.go:122] "RemoveContainer" containerID="ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f"
Nov 03 11:22:34.837682 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:34.837660    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f\": container with ID starting with ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f not found: ID does not exist" containerID="ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f"
Nov 03 11:22:34.837771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.837689    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f"} err="failed to get container status \"ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f\": rpc error: code = NotFound desc = could not find container \"ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f\": container with ID starting with ddd1f01b66f8bdb99cf85f0b416df17ca831e276d2b17099fe0c2137fd573b8f not found: ID does not exist"
Nov 03 11:22:34.845933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.845892    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr" podStartSLOduration=1.8458793839999998 podStartE2EDuration="1.845879384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:22:34.840025011 +0000 UTC m=+3185.045607060" watchObservedRunningTime="2026-11-03 11:22:34.845879384 +0000 UTC m=+3185.051461477"
Nov 03 11:22:34.858193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.858165    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"]
Nov 03 11:22:34.863950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.863927    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-rx2hq"]
Nov 03 11:22:34.999597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:34.999482    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:35.135180 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:35.135140    5817 scope.go:122] "RemoveContainer" containerID="83bd7978a916e6713bb5f30cc73efda4f2debf5d7f77fee1ef4fea58a6c1fa9b"
Nov 03 11:22:35.148795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:35.148775    5817 scope.go:122] "RemoveContainer" containerID="54ca5f8d01f229c78eba31786752fea9bafa221b44102b63c0f6635f25dc9988"
Nov 03 11:22:35.161003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:35.160975    5817 scope.go:122] "RemoveContainer" containerID="9d5f1c3646198d0b3b6300021023d80653d800597d90a56cea2da3627149e31a"
Nov 03 11:22:35.171339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:35.171318    5817 scope.go:122] "RemoveContainer" containerID="1304877d6df360e85279a1d172cb92cd4fb16881f2bf2a279f2546feea311863"
Nov 03 11:22:35.683364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:35.681295    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:35.837300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:35.837261    5817 generic.go:356] "Generic (PLEG): container finished" podID="aa059d4c-5a2f-461d-a414-3ca128998417" containerID="e4ab3554cec4f2678797142b16e3873af8a60a05df2dd7ada8729796ac859ab8" exitCode=0
Nov 03 11:22:35.837475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:35.837357    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr" event={"ID":"aa059d4c-5a2f-461d-a414-3ca128998417","Type":"ContainerDied","Data":"e4ab3554cec4f2678797142b16e3873af8a60a05df2dd7ada8729796ac859ab8"}
Nov 03 11:22:35.904534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:35.904503    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" path="/var/lib/kubelet/pods/aa54aa54-6c59-4169-a0f2-beb2ca32557b/volumes"
Nov 03 11:22:36.477550 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:36.477512    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:37.093593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.060499    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:37.111671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.109824    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:37.241128 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.241091    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa059d4c-5a2f-461d-a414-3ca128998417-kube-api-access-fl464\" (UniqueName: \"kubernetes.io/projected/aa059d4c-5a2f-461d-a414-3ca128998417-kube-api-access-fl464\") pod \"aa059d4c-5a2f-461d-a414-3ca128998417\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") "
Nov 03 11:22:37.241305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.241290    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-xks-ca\") pod \"aa059d4c-5a2f-461d-a414-3ca128998417\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") "
Nov 03 11:22:37.241381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.241366    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-openshift-ca\") pod \"aa059d4c-5a2f-461d-a414-3ca128998417\" (UID: \"aa059d4c-5a2f-461d-a414-3ca128998417\") "
Nov 03 11:22:37.241780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.241751    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-openshift-ca" pod "aa059d4c-5a2f-461d-a414-3ca128998417" (UID: "aa059d4c-5a2f-461d-a414-3ca128998417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:37.242513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.242489    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:37.243663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.243635    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-xks-ca" pod "aa059d4c-5a2f-461d-a414-3ca128998417" (UID: "aa059d4c-5a2f-461d-a414-3ca128998417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:37.243794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.243675    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa059d4c-5a2f-461d-a414-3ca128998417-kube-api-access-fl464" pod "aa059d4c-5a2f-461d-a414-3ca128998417" (UID: "aa059d4c-5a2f-461d-a414-3ca128998417"). InnerVolumeSpecName "kube-api-access-fl464". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:37.344018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.343988    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl464\" (UniqueName: \"kubernetes.io/projected/aa059d4c-5a2f-461d-a414-3ca128998417-kube-api-access-fl464\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:37.344018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.344014    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa059d4c-5a2f-461d-a414-3ca128998417-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:37.934328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.928223    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:37.965275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.965249    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"
Nov 03 11:22:37.966872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.966844    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr" event={"ID":"aa059d4c-5a2f-461d-a414-3ca128998417","Type":"ContainerDied","Data":"009202a6a59f0a7ed915b708b57dca27c1de910ef41af49cc1dbf83ae9c65ea5"}
Nov 03 11:22:37.966994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:37.966894    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="009202a6a59f0a7ed915b708b57dca27c1de910ef41af49cc1dbf83ae9c65ea5"
Nov 03 11:22:38.465920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:38.465886    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:38.611634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:38.606519    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:38.931599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:38.931504    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:38.969597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:38.955233    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"]
Nov 03 11:22:39.009407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.009375    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3eb43f-tdw28"]
Nov 03 11:22:39.133917 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.133415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:39.355742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.355719    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:22:39.405789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.405756    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f3810a-c5ff-41bf-bd62-8767c026017b-kube-api-access-4jwst\" (UniqueName: \"kubernetes.io/projected/b7f3810a-c5ff-41bf-bd62-8767c026017b-kube-api-access-4jwst\") pod \"b7f3810a-c5ff-41bf-bd62-8767c026017b\" (UID: \"b7f3810a-c5ff-41bf-bd62-8767c026017b\") "
Nov 03 11:22:39.406051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.406013    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f3810a-c5ff-41bf-bd62-8767c026017b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f3810a-c5ff-41bf-bd62-8767c026017b-plugins-config-volume\") pod \"b7f3810a-c5ff-41bf-bd62-8767c026017b\" (UID: \"b7f3810a-c5ff-41bf-bd62-8767c026017b\") "
Nov 03 11:22:39.407836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.407809    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f3810a-c5ff-41bf-bd62-8767c026017b-plugins-config-volume" pod "b7f3810a-c5ff-41bf-bd62-8767c026017b" (UID: "b7f3810a-c5ff-41bf-bd62-8767c026017b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:39.408680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.408652    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f3810a-c5ff-41bf-bd62-8767c026017b-kube-api-access-4jwst" pod "b7f3810a-c5ff-41bf-bd62-8767c026017b" (UID: "b7f3810a-c5ff-41bf-bd62-8767c026017b"). InnerVolumeSpecName "kube-api-access-4jwst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:39.494875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.494803    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:39.509327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.509301    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwst\" (UniqueName: \"kubernetes.io/projected/b7f3810a-c5ff-41bf-bd62-8767c026017b-kube-api-access-4jwst\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:39.509327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.509325    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f3810a-c5ff-41bf-bd62-8767c026017b-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:39.855823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.854528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:39.917657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:39.917622    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f34536-d074-41af-9c44-187d5e75f938" path="/var/lib/kubelet/pods/f4f34536-d074-41af-9c44-187d5e75f938/volumes"
Nov 03 11:22:40.047612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.047565    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:40.050284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.050262    5817 generic.go:356] "Generic (PLEG): container finished" podID="b7f3810a-c5ff-41bf-bd62-8767c026017b" containerID="a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24" exitCode=0
Nov 03 11:22:40.050404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.050308    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"
Nov 03 11:22:40.050404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.050343    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk" event={"ID":"b7f3810a-c5ff-41bf-bd62-8767c026017b","Type":"ContainerDied","Data":"a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24"}
Nov 03 11:22:40.050404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.050396    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk" event={"ID":"b7f3810a-c5ff-41bf-bd62-8767c026017b","Type":"ContainerDied","Data":"2fe4cc0e5eceab9ce81425bda301f5eafa00563461458d707c1dc068f6d23a9c"}
Nov 03 11:22:40.050514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.050429    5817 scope.go:122] "RemoveContainer" containerID="a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24"
Nov 03 11:22:40.062529 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.062512    5817 scope.go:122] "RemoveContainer" containerID="a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24"
Nov 03 11:22:40.062861 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:40.062839    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24\": container with ID starting with a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24 not found: ID does not exist" containerID="a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24"
Nov 03 11:22:40.062939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.062871    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24"} err="failed to get container status \"a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24\": rpc error: code = NotFound desc = could not find container \"a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24\": container with ID starting with a9b8a9d88c2cf8baf37a35032547a69b6137a46202bca419e6c1a7aab8d63c24 not found: ID does not exist"
Nov 03 11:22:40.072401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.072379    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"]
Nov 03 11:22:40.077582 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.077549    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24376f-7c65966447-4bkpk"]
Nov 03 11:22:40.489140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:40.489106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:41.153605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:41.152593    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:41.800607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:41.799485    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:41.907872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:41.907836    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f3810a-c5ff-41bf-bd62-8767c026017b" path="/var/lib/kubelet/pods/b7f3810a-c5ff-41bf-bd62-8767c026017b/volumes"
Nov 03 11:22:42.566729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:42.561687    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:43.184779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:43.184745    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:43.830390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:43.830352    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:44.023255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.023217    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:44.339898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.339862    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"]
Nov 03 11:22:44.340732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340715    5817 cpu_manager.go:414] "containerMap: removing container" podUID="aa059d4c-5a2f-461d-a414-3ca128998417" containerName="revoke-keys"
Nov 03 11:22:44.340732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340732    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa059d4c-5a2f-461d-a414-3ca128998417" containerName="revoke-keys"
Nov 03 11:22:44.340838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340750    5817 cpu_manager.go:414] "containerMap: removing container" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy"
Nov 03 11:22:44.340838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340755    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy"
Nov 03 11:22:44.340838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340782    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b7f3810a-c5ff-41bf-bd62-8767c026017b" containerName="payload-pre-processing"
Nov 03 11:22:44.340838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340788    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f3810a-c5ff-41bf-bd62-8767c026017b" containerName="payload-pre-processing"
Nov 03 11:22:44.340987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340938    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa059d4c-5a2f-461d-a414-3ca128998417" containerName="revoke-keys"
Nov 03 11:22:44.340987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340948    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa54aa54-6c59-4169-a0f2-beb2ca32557b" containerName="istio-proxy"
Nov 03 11:22:44.340987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.340957    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f3810a-c5ff-41bf-bd62-8767c026017b" containerName="payload-pre-processing"
Nov 03 11:22:44.346435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.346410    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.350017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.349993    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:22:44.354950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.354767    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"]
Nov 03 11:22:44.355071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.354990    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:22:44.484929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.484896    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twj2b\" (UniqueName: \"kubernetes.io/projected/629e49ef-8016-461e-9eba-7ab15bab0e31-kube-api-access-twj2b\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.485154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.485070    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.485272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.485253    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.586475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.586445    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.586671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.586560    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.586827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.586806    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twj2b\" (UniqueName: \"kubernetes.io/projected/629e49ef-8016-461e-9eba-7ab15bab0e31-kube-api-access-twj2b\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.587250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.587190    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.589126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.589097    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.595029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.594964    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twj2b\" (UniqueName: \"kubernetes.io/projected/629e49ef-8016-461e-9eba-7ab15bab0e31-kube-api-access-twj2b\") pod \"maas-api-revoke-sub-1b326624-s8dnd\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.671875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.671846    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:44.878669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.872194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:44.973942 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:22:44.973913    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629e49ef_8016_461e_9eba_7ab15bab0e31.slice/crio-ba7b50d7be665731dc911c17c4f2e13a093491edae38c5d0a4741af10120e29c WatchSource:0}: Error finding container ba7b50d7be665731dc911c17c4f2e13a093491edae38c5d0a4741af10120e29c: Status 404 returned error can't find the container with id ba7b50d7be665731dc911c17c4f2e13a093491edae38c5d0a4741af10120e29c
Nov 03 11:22:44.974873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:44.974849    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"]
Nov 03 11:22:45.123563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:45.123527    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd" event={"ID":"629e49ef-8016-461e-9eba-7ab15bab0e31","Type":"ContainerStarted","Data":"ba7b50d7be665731dc911c17c4f2e13a093491edae38c5d0a4741af10120e29c"}
Nov 03 11:22:45.490597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:45.490479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:45.702923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:45.702876    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:45.835095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:45.835003    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:46.136705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:46.136632    5817 generic.go:356] "Generic (PLEG): container finished" podID="629e49ef-8016-461e-9eba-7ab15bab0e31" containerID="a208708dba75885734d79033cc360afba1fec576cf42bbbba093e1b596ad7910" exitCode=0
Nov 03 11:22:46.137072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:46.136721    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd" event={"ID":"629e49ef-8016-461e-9eba-7ab15bab0e31","Type":"ContainerDied","Data":"a208708dba75885734d79033cc360afba1fec576cf42bbbba093e1b596ad7910"}
Nov 03 11:22:46.283592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:46.281048    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:46.581016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:46.580986    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:46.964597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:46.947795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:47.293680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.293598    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"]
Nov 03 11:22:47.294106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.293874    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy" containerID="cri-o://2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b" gracePeriod=30
Nov 03 11:22:47.353147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.353123    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:47.375480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.375455    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-xks-ca\") pod \"629e49ef-8016-461e-9eba-7ab15bab0e31\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") "
Nov 03 11:22:47.375645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.375583    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629e49ef-8016-461e-9eba-7ab15bab0e31-kube-api-access-twj2b\" (UniqueName: \"kubernetes.io/projected/629e49ef-8016-461e-9eba-7ab15bab0e31-kube-api-access-twj2b\") pod \"629e49ef-8016-461e-9eba-7ab15bab0e31\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") "
Nov 03 11:22:47.375940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.375912    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-openshift-ca\") pod \"629e49ef-8016-461e-9eba-7ab15bab0e31\" (UID: \"629e49ef-8016-461e-9eba-7ab15bab0e31\") "
Nov 03 11:22:47.376323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.376299    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-openshift-ca" pod "629e49ef-8016-461e-9eba-7ab15bab0e31" (UID: "629e49ef-8016-461e-9eba-7ab15bab0e31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:47.376772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.376750    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:47.377906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.377879    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629e49ef-8016-461e-9eba-7ab15bab0e31-kube-api-access-twj2b" pod "629e49ef-8016-461e-9eba-7ab15bab0e31" (UID: "629e49ef-8016-461e-9eba-7ab15bab0e31"). InnerVolumeSpecName "kube-api-access-twj2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:47.378008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.377920    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-xks-ca" pod "629e49ef-8016-461e-9eba-7ab15bab0e31" (UID: "629e49ef-8016-461e-9eba-7ab15bab0e31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:47.478838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.478804    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629e49ef-8016-461e-9eba-7ab15bab0e31-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:47.479043 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.478931    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twj2b\" (UniqueName: \"kubernetes.io/projected/629e49ef-8016-461e-9eba-7ab15bab0e31-kube-api-access-twj2b\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:47.726657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:47.726531    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:48.157205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:48.157124    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"
Nov 03 11:22:48.157205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:48.157147    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd" event={"ID":"629e49ef-8016-461e-9eba-7ab15bab0e31","Type":"ContainerDied","Data":"ba7b50d7be665731dc911c17c4f2e13a093491edae38c5d0a4741af10120e29c"}
Nov 03 11:22:48.157444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:48.157231    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba7b50d7be665731dc911c17c4f2e13a093491edae38c5d0a4741af10120e29c"
Nov 03 11:22:48.492154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:48.492106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:49.342598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:49.341275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:49.983065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:49.983030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:50.082264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:50.082234    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"]
Nov 03 11:22:50.082453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:50.082432    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy" containerID="cri-o://c5b636f3ee37f1b07c8c7f55802f2fd9f9b7b524e8a85940db2c849b50f6c7df" gracePeriod=30
Nov 03 11:22:50.769602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:50.766639    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:51.459601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:51.449799    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:52.210595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.210367    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:52.416425 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:52.416175    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:22:52.616497 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.616469    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 11:22:52.726278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.726246    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-socket\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.726602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.726563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-credential-socket\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.726820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.726801    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-istio-token\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-istio-token\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.726938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.726597    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-socket" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:52.727005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.726910    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-credential-socket" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:52.727108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.727092    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc989773-c123-455c-94e3-8cc103087d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc989773-c123-455c-94e3-8cc103087d84-istio-podinfo\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.727250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.727234    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-kube-api-access-khr6r\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-kube-api-access-khr6r\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.727419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.727389    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-envoy\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.728084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.728047    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-data" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:52.728258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.728243    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-data\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.729077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.728456    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc989773-c123-455c-94e3-8cc103087d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc989773-c123-455c-94e3-8cc103087d84-istiod-ca-cert\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.729291 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.729023    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc989773-c123-455c-94e3-8cc103087d84-istiod-ca-cert" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:52.729534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.729516    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-certs\") pod \"cc989773-c123-455c-94e3-8cc103087d84\" (UID: \"cc989773-c123-455c-94e3-8cc103087d84\") "
Nov 03 11:22:52.730131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.729907    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-certs" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:52.731191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.731169    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:52.731330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.731315    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc989773-c123-455c-94e3-8cc103087d84-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:52.731439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.731425    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:52.731537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.731524    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:52.731694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.731679    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:52.736440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.736206    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc989773-c123-455c-94e3-8cc103087d84-istio-podinfo" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:52.738850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.738803    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-envoy" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:52.740838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.740792    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-istio-token" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:52.748809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.748779    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-kube-api-access-khr6r" pod "cc989773-c123-455c-94e3-8cc103087d84" (UID: "cc989773-c123-455c-94e3-8cc103087d84"). InnerVolumeSpecName "kube-api-access-khr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:52.847321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.847215    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:52.847321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.847253    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc989773-c123-455c-94e3-8cc103087d84-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:52.847321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.847268    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khr6r\" (UniqueName: \"kubernetes.io/projected/cc989773-c123-455c-94e3-8cc103087d84-kube-api-access-khr6r\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:52.847321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:52.847284    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc989773-c123-455c-94e3-8cc103087d84-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:53.166181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.166088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:53.263616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.259710    5817 generic.go:356] "Generic (PLEG): container finished" podID="cc989773-c123-455c-94e3-8cc103087d84" containerID="2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b" exitCode=0
Nov 03 11:22:53.263616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.259760    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"
Nov 03 11:22:53.263616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.259808    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" event={"ID":"cc989773-c123-455c-94e3-8cc103087d84","Type":"ContainerDied","Data":"2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b"}
Nov 03 11:22:53.263616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.259874    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb" event={"ID":"cc989773-c123-455c-94e3-8cc103087d84","Type":"ContainerDied","Data":"6af76f80ba2ac94b23bc05a26927b7685b24b4456a14be0c124e60fb3f401607"}
Nov 03 11:22:53.263616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.259918    5817 scope.go:122] "RemoveContainer" containerID="2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b"
Nov 03 11:22:53.279287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.279255    5817 scope.go:122] "RemoveContainer" containerID="2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b"
Nov 03 11:22:53.279657 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:22:53.279607    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b\": container with ID starting with 2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b not found: ID does not exist" containerID="2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b"
Nov 03 11:22:53.279657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.279644    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b"} err="failed to get container status \"2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b\": rpc error: code = NotFound desc = could not find container \"2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b\": container with ID starting with 2f0f5081e0bfc6aa35041f174e1262f973d8a88c5c9431956cbae41877d7421b not found: ID does not exist"
Nov 03 11:22:53.331722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.331678    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"]
Nov 03 11:22:53.347856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.347820    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-g95wb"]
Nov 03 11:22:53.707460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.707423    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:53.907696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.907663    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc989773-c123-455c-94e3-8cc103087d84" path="/var/lib/kubelet/pods/cc989773-c123-455c-94e3-8cc103087d84/volumes"
Nov 03 11:22:53.913093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:53.913055    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:54.369791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:54.369760    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"]
Nov 03 11:22:54.369992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:54.369971    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy" containerID="cri-o://2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05" gracePeriod=30
Nov 03 11:22:54.525488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:54.525448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:54.666183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:54.666090    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:55.005141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.004239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:55.297623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.279765    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"]
Nov 03 11:22:55.297623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.281040    5817 cpu_manager.go:414] "containerMap: removing container" podUID="629e49ef-8016-461e-9eba-7ab15bab0e31" containerName="revoke-keys"
Nov 03 11:22:55.297623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.281061    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629e49ef-8016-461e-9eba-7ab15bab0e31" containerName="revoke-keys"
Nov 03 11:22:55.297623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.281132    5817 cpu_manager.go:414] "containerMap: removing container" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy"
Nov 03 11:22:55.297623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.281141    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy"
Nov 03 11:22:55.297623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.281357    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc989773-c123-455c-94e3-8cc103087d84" containerName="istio-proxy"
Nov 03 11:22:55.297623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.281387    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="629e49ef-8016-461e-9eba-7ab15bab0e31" containerName="revoke-keys"
Nov 03 11:22:55.297623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.289286    5817 generic.go:356] "Generic (PLEG): container finished" podID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerID="c5b636f3ee37f1b07c8c7f55802f2fd9f9b7b524e8a85940db2c849b50f6c7df" exitCode=0
Nov 03 11:22:55.314632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.311761    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"]
Nov 03 11:22:55.314632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.311806    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" event={"ID":"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a","Type":"ContainerDied","Data":"c5b636f3ee37f1b07c8c7f55802f2fd9f9b7b524e8a85940db2c849b50f6c7df"}
Nov 03 11:22:55.314632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.312695    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.318674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.318648    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:22:55.318812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.318739    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:22:55.347753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.347720    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.347883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.347780    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.347927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.347908    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vwm\" (UniqueName: \"kubernetes.io/projected/79d6bb17-b7d8-47e8-a6a6-056c46360439-kube-api-access-p7vwm\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.412913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.412889    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:22:55.449023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.448988    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-podinfo\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449089    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-envoy\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449221    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-socket\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449252    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-token\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-token\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449355    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-credential-socket\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449413    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-certs\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449442    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-data\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449592    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istiod-ca-cert\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449646    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-kube-api-access-5hxnr\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-kube-api-access-5hxnr\") pod \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\" (UID: \"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a\") "
Nov 03 11:22:55.449664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449641    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-credential-socket" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:55.449805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449786    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-certs" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:55.450029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.449998    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vwm\" (UniqueName: \"kubernetes.io/projected/79d6bb17-b7d8-47e8-a6a6-056c46360439-kube-api-access-p7vwm\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.450154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.450117    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-data" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:55.450507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.450393    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.450507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.450470    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.451000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.450706    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istiod-ca-cert" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:55.451307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.451281    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.451375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.451313    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.451375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.451328    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.451375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.451341    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.452096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.452071    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.453122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.453098    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-envoy" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:55.453836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.453806    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-socket" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:55.454457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.454431    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.454661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.454638    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-podinfo" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:55.456086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.456043    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-kube-api-access-5hxnr" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "kube-api-access-5hxnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:55.458523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.458496    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vwm\" (UniqueName: \"kubernetes.io/projected/79d6bb17-b7d8-47e8-a6a6-056c46360439-kube-api-access-p7vwm\") pod \"maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.468927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.468903    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-token" pod "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" (UID: "d3b519ba-f9f9-46a4-b91a-1dc194d80f6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:55.553899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.553800    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.553899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.553834    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.553899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.553848    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.553899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.553861    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.553899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.553877    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hxnr\" (UniqueName: \"kubernetes.io/projected/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a-kube-api-access-5hxnr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:55.636453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.636417    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:55.759121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.756790    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:55.929340 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.929310    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"]
Nov 03 11:22:55.929525 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:22:55.929500    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d6bb17_b7d8_47e8_a6a6_056c46360439.slice/crio-a26b669b98038c12f8cd6bd3abb3abe8175400d6d6fa735c13a5ff776a9aa3da WatchSource:0}: Error finding container a26b669b98038c12f8cd6bd3abb3abe8175400d6d6fa735c13a5ff776a9aa3da: Status 404 returned error can't find the container with id a26b669b98038c12f8cd6bd3abb3abe8175400d6d6fa735c13a5ff776a9aa3da
Nov 03 11:22:55.990135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.990108    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"]
Nov 03 11:22:55.990673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:55.990641    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy" containerID="cri-o://731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976" gracePeriod=30
Nov 03 11:22:56.053545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.053514    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:56.368282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.368241    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq" event={"ID":"d3b519ba-f9f9-46a4-b91a-1dc194d80f6a","Type":"ContainerDied","Data":"b94e4a521565a9379430e60b57ca8dd9aa9a04e51ac437047417cc27392550f7"}
Nov 03 11:22:56.368487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.368318    5817 scope.go:122] "RemoveContainer" containerID="c5b636f3ee37f1b07c8c7f55802f2fd9f9b7b524e8a85940db2c849b50f6c7df"
Nov 03 11:22:56.368487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.368464    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"
Nov 03 11:22:56.376910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.376840    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd" event={"ID":"79d6bb17-b7d8-47e8-a6a6-056c46360439","Type":"ContainerStarted","Data":"a26b669b98038c12f8cd6bd3abb3abe8175400d6d6fa735c13a5ff776a9aa3da"}
Nov 03 11:22:56.408785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.406441    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"]
Nov 03 11:22:56.413298 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.413272    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-tgztq"]
Nov 03 11:22:56.745600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.741482    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:56.957269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:56.957238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:57.088534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.088438    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:57.388373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.388286    5817 generic.go:356] "Generic (PLEG): container finished" podID="79d6bb17-b7d8-47e8-a6a6-056c46360439" containerID="55d24b5a221ec682f0c811d95b602cda200d2446f209da877454889415f93bfb" exitCode=0
Nov 03 11:22:57.388554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.388368    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd" event={"ID":"79d6bb17-b7d8-47e8-a6a6-056c46360439","Type":"ContainerDied","Data":"55d24b5a221ec682f0c811d95b602cda200d2446f209da877454889415f93bfb"}
Nov 03 11:22:57.557075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.557040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:57.750322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.750284    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"]
Nov 03 11:22:57.751259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.751236    5817 cpu_manager.go:414] "containerMap: removing container" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy"
Nov 03 11:22:57.751259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.751259    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy"
Nov 03 11:22:57.751463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.751438    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" containerName="istio-proxy"
Nov 03 11:22:57.755440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.755419    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.765732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.765707    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"]
Nov 03 11:22:57.838727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.838686    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.839554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.838797    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8f6f\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-kube-api-access-l8f6f\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.839554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.838888    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.839554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.838922    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.839554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.838948    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.839554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.838981    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.840526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.839686    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.840526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.839786    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.840526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.839865    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.908429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.908395    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b519ba-f9f9-46a4-b91a-1dc194d80f6a" path="/var/lib/kubelet/pods/d3b519ba-f9f9-46a4-b91a-1dc194d80f6a/volumes"
Nov 03 11:22:57.941460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.941420    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.941663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.941477    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.941663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.941502    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.941663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.941535    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.941810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.941730    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.941864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.941849    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.941944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.941930    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.942009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.941995    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.942112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.942098    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8f6f\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-kube-api-access-l8f6f\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.943539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.943401    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.943710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.943694    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.943945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.943920    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.947813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.945056    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.947813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.945281    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.948277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.948252    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.948812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.948725    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.955859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.955837    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8f6f\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-kube-api-access-l8f6f\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.968464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.967027    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:57.977801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:57.977701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:58.076480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.076401    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:22:58.130388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.130355    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:58.382134 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:22:58.382097    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcbfc0b_0e4b_4a69_bf0c_3f0065308b4a.slice/crio-3f17cd783f3a9d8e9870cb6c5073cdf261a73ec62e18f7fa829f80e73fe6488f WatchSource:0}: Error finding container 3f17cd783f3a9d8e9870cb6c5073cdf261a73ec62e18f7fa829f80e73fe6488f: Status 404 returned error can't find the container with id 3f17cd783f3a9d8e9870cb6c5073cdf261a73ec62e18f7fa829f80e73fe6488f
Nov 03 11:22:58.384561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.384540    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"]
Nov 03 11:22:58.386166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.386129    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:58.386270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.386213    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:58.386270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.386263    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:58.406653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.406498    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" event={"ID":"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a","Type":"ContainerStarted","Data":"3f17cd783f3a9d8e9870cb6c5073cdf261a73ec62e18f7fa829f80e73fe6488f"}
Nov 03 11:22:58.477975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.470084    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:58.682621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.682564    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:58.764358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.764329    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-openshift-ca\") pod \"79d6bb17-b7d8-47e8-a6a6-056c46360439\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") "
Nov 03 11:22:58.764548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.764446    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-xks-ca\") pod \"79d6bb17-b7d8-47e8-a6a6-056c46360439\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") "
Nov 03 11:22:58.764678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.764661    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d6bb17-b7d8-47e8-a6a6-056c46360439-kube-api-access-p7vwm\" (UniqueName: \"kubernetes.io/projected/79d6bb17-b7d8-47e8-a6a6-056c46360439-kube-api-access-p7vwm\") pod \"79d6bb17-b7d8-47e8-a6a6-056c46360439\" (UID: \"79d6bb17-b7d8-47e8-a6a6-056c46360439\") "
Nov 03 11:22:58.764754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.764726    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-openshift-ca" pod "79d6bb17-b7d8-47e8-a6a6-056c46360439" (UID: "79d6bb17-b7d8-47e8-a6a6-056c46360439"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:58.765257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.765234    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:58.766633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.766605    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-xks-ca" pod "79d6bb17-b7d8-47e8-a6a6-056c46360439" (UID: "79d6bb17-b7d8-47e8-a6a6-056c46360439"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:58.766920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.766901    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d6bb17-b7d8-47e8-a6a6-056c46360439-kube-api-access-p7vwm" pod "79d6bb17-b7d8-47e8-a6a6-056c46360439" (UID: "79d6bb17-b7d8-47e8-a6a6-056c46360439"). InnerVolumeSpecName "kube-api-access-p7vwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:58.867181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.867140    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7vwm\" (UniqueName: \"kubernetes.io/projected/79d6bb17-b7d8-47e8-a6a6-056c46360439-kube-api-access-p7vwm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:58.867181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:58.867177    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d6bb17-b7d8-47e8-a6a6-056c46360439-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.228926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.222254    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:22:59.420924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.420872    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"
Nov 03 11:22:59.421155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.421012    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd" event={"ID":"79d6bb17-b7d8-47e8-a6a6-056c46360439","Type":"ContainerDied","Data":"a26b669b98038c12f8cd6bd3abb3abe8175400d6d6fa735c13a5ff776a9aa3da"}
Nov 03 11:22:59.421155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.421063    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a26b669b98038c12f8cd6bd3abb3abe8175400d6d6fa735c13a5ff776a9aa3da"
Nov 03 11:22:59.426443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.426415    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" event={"ID":"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a","Type":"ContainerStarted","Data":"2f6b37cf17f6b81d5f8bd5b593193b83929f1433c86b6a16446e1f4f00115985"}
Nov 03 11:22:59.464616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.464503    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" podStartSLOduration=2.464482539 podStartE2EDuration="2.464482539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:22:59.459696019 +0000 UTC m=+3209.665278070" watchObservedRunningTime="2026-11-03 11:22:59.464482539 +0000 UTC m=+3209.670064592"
Nov 03 11:22:59.765440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.765415    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823184    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istiod-ca-cert\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823367    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-socket\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823398    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-kube-api-access-cr7lf\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-kube-api-access-cr7lf\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823449    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-certs\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823514    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-data\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823547    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-token\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-token\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823701    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-podinfo\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823782    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-credential-socket\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.828597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.823853    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-envoy\") pod \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\" (UID: \"a9b38bf7-aad0-494a-9e2a-13dd62b782c7\") "
Nov 03 11:22:59.849606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.831010    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istiod-ca-cert" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:59.849606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.834046    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-socket" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.849606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.834375    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-data" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.849606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.834607    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-credential-socket" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.849606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.842682    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-certs" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.863184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.856683    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-token" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:59.863184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.863126    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-kube-api-access-cr7lf" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "kube-api-access-cr7lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:59.865507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.865465    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-envoy" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.870072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.870043    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-podinfo" pod "a9b38bf7-aad0-494a-9e2a-13dd62b782c7" (UID: "a9b38bf7-aad0-494a-9e2a-13dd62b782c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947412    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947451    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947467    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr7lf\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-kube-api-access-cr7lf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947484    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947500    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947513    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947525    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947539    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.947826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:22:59.947552    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b38bf7-aad0-494a-9e2a-13dd62b782c7-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:00.042022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.041043    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:00.089602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.088698    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:23:00.105594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.103121    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 11:23:00.105594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.103178    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 11:23:00.439044 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.439012    5817 generic.go:356] "Generic (PLEG): container finished" podID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerID="2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05" exitCode=0
Nov 03 11:23:00.439548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.439104    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" event={"ID":"a9b38bf7-aad0-494a-9e2a-13dd62b782c7","Type":"ContainerDied","Data":"2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05"}
Nov 03 11:23:00.439548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.439126    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"
Nov 03 11:23:00.439548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.439162    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6" event={"ID":"a9b38bf7-aad0-494a-9e2a-13dd62b782c7","Type":"ContainerDied","Data":"6714105c3bfe0f360aa36e9897ab2eb32f86e2f32aa02af56e15b469d23aaff3"}
Nov 03 11:23:00.439548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.439176    5817 scope.go:122] "RemoveContainer" containerID="2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05"
Nov 03 11:23:00.453035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.453013    5817 scope.go:122] "RemoveContainer" containerID="2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05"
Nov 03 11:23:00.453391 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:00.453363    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05\": container with ID starting with 2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05 not found: ID does not exist" containerID="2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05"
Nov 03 11:23:00.453492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.453400    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05"} err="failed to get container status \"2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05\": rpc error: code = NotFound desc = could not find container \"2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05\": container with ID starting with 2a5285abf873d8afd0c914ba5f15ce01326a468e30cf01350b05a13e2254ed05 not found: ID does not exist"
Nov 03 11:23:00.460866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.460831    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"]
Nov 03 11:23:00.465932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.465911    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-8s4w6"]
Nov 03 11:23:00.745457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:00.745335    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:01.077541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.077508    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 11:23:01.077727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.077556    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 11:23:01.369116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.369091    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:23:01.460436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.460401    5817 generic.go:356] "Generic (PLEG): container finished" podID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerID="731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976" exitCode=0
Nov 03 11:23:01.460996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.460457    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"
Nov 03 11:23:01.460996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.460491    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" event={"ID":"dea678e2-cfc3-4ff5-8889-601be2dd2909","Type":"ContainerDied","Data":"731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976"}
Nov 03 11:23:01.460996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.460558    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss" event={"ID":"dea678e2-cfc3-4ff5-8889-601be2dd2909","Type":"ContainerDied","Data":"417e801047465a701090828d11f649380be937d823aa56fe215526a0d68b3b76"}
Nov 03 11:23:01.460996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.460594    5817 scope.go:122] "RemoveContainer" containerID="731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976"
Nov 03 11:23:01.501597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.499682    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.531684    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-data\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.531759    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-token\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-token\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.531851    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-kube-api-access-25zjm\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-kube-api-access-25zjm\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.531907    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-certs\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.532001    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-envoy\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.532056    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea678e2-cfc3-4ff5-8889-601be2dd2909-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea678e2-cfc3-4ff5-8889-601be2dd2909-istiod-ca-cert\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.532111    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-credential-socket\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.532194    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-podinfo\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.537031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.532265    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-socket\") pod \"dea678e2-cfc3-4ff5-8889-601be2dd2909\" (UID: \"dea678e2-cfc3-4ff5-8889-601be2dd2909\") "
Nov 03 11:23:01.548226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.542651    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-certs" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:01.548226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.545748    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-data" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:01.561192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.560890    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-credential-socket" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:01.569871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.561527    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea678e2-cfc3-4ff5-8889-601be2dd2909-istiod-ca-cert" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:01.571500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.571467    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea678e2-cfc3-4ff5-8889-601be2dd2909-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.571692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.571505    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.571692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.571519    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.571692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.571535    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.574384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.573895    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-socket" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:01.604627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.594344    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-kube-api-access-25zjm" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "kube-api-access-25zjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:01.604627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.594438    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-podinfo" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:23:01.604627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.595841    5817 scope.go:122] "RemoveContainer" containerID="e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786"
Nov 03 11:23:01.637180 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.616727    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-envoy" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:01.637841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.637750    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-token" pod "dea678e2-cfc3-4ff5-8889-601be2dd2909" (UID: "dea678e2-cfc3-4ff5-8889-601be2dd2909"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:01.656227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.656205    5817 scope.go:122] "RemoveContainer" containerID="731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976"
Nov 03 11:23:01.656607 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:01.656535    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976\": container with ID starting with 731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976 not found: ID does not exist" containerID="731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976"
Nov 03 11:23:01.656716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.656615    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976"} err="failed to get container status \"731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976\": rpc error: code = NotFound desc = could not find container \"731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976\": container with ID starting with 731302f247b7501c296db0a57978567404bfd1d61a3e01e14793f524618f3976 not found: ID does not exist"
Nov 03 11:23:01.656716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.656638    5817 scope.go:122] "RemoveContainer" containerID="e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786"
Nov 03 11:23:01.656975 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:01.656938    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786\": container with ID starting with e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786 not found: ID does not exist" containerID="e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786"
Nov 03 11:23:01.657037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.656981    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786"} err="failed to get container status \"e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786\": rpc error: code = NotFound desc = could not find container \"e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786\": container with ID starting with e3c48666481ec6cecc8fffcacb0e97b87eb707636d1b74e6dd7460234c51b786 not found: ID does not exist"
Nov 03 11:23:01.673452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.673385    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.673452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.673407    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.673452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.673418    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25zjm\" (UniqueName: \"kubernetes.io/projected/dea678e2-cfc3-4ff5-8889-601be2dd2909-kube-api-access-25zjm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.673452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.673426    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.673452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.673434    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea678e2-cfc3-4ff5-8889-601be2dd2909-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:01.786341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.786307    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"]
Nov 03 11:23:01.821722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.821373    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3eb43f-openshift-default-5ff498f587-hf4ss"]
Nov 03 11:23:01.929819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.929776    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" path="/var/lib/kubelet/pods/a9b38bf7-aad0-494a-9e2a-13dd62b782c7/volumes"
Nov 03 11:23:01.930517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:01.930495    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" path="/var/lib/kubelet/pods/dea678e2-cfc3-4ff5-8889-601be2dd2909/volumes"
Nov 03 11:23:02.076911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:02.076883    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 11:23:02.077097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:02.076939    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 11:23:02.220312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:02.220224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:02.983599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:02.981378    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:03.079609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:03.078885    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 11:23:03.079609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:03.078942    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 11:23:03.668335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:03.668301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:04.076807 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:04.076777    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 11:23:04.077230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:04.076824    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 11:23:04.280628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:04.280090    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:04.928599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:04.921403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:05.076684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:05.076649    5817 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 11:23:05.076888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:05.076701    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 11:23:05.512034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:05.511989    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:05.654608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:05.654194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:06.082127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:06.082098    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:23:06.082647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:06.082585    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:23:06.083400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:06.083379    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:23:06.286789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:06.286749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:06.806488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:06.806459    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:06.964143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:06.964108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:07.159524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:07.159428    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:07.283849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:07.283771    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"]
Nov 03 11:23:07.313513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:07.313478    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-68aa46-9sndd"]
Nov 03 11:23:07.443601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:07.438484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:07.728887 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:07.728789    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:07.875384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:07.875348    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:07.906563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:07.906528    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d6bb17-b7d8-47e8-a6a6-056c46360439" path="/var/lib/kubelet/pods/79d6bb17-b7d8-47e8-a6a6-056c46360439/volumes"
Nov 03 11:23:08.160675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:08.158136    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:08.444672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:08.444636    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:08.699055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:08.698973    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:08.837757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:08.837721    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:09.472101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:09.472064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:10.160699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:10.160666    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:11.061250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:11.061211    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:11.741109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:11.741077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:12.350599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:12.349365    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:12.997605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:12.996822    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:13.729014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:13.728967    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:14.263933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:14.263742    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:14.451056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:14.451015    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:14.477227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:14.477192    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"]
Nov 03 11:23:14.477499 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:14.477470    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy" containerID="cri-o://fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6" gracePeriod=30
Nov 03 11:23:14.633646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:14.629241    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:15.104997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.104963    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:15.128514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.128486    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"]
Nov 03 11:23:15.129320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129304    5817 cpu_manager.go:414] "containerMap: removing container" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy"
Nov 03 11:23:15.129320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129321    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy"
Nov 03 11:23:15.129431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129344    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy"
Nov 03 11:23:15.129431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129349    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy"
Nov 03 11:23:15.129431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129376    5817 cpu_manager.go:414] "containerMap: removing container" podUID="79d6bb17-b7d8-47e8-a6a6-056c46360439" containerName="revoke-keys"
Nov 03 11:23:15.129431 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129381    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d6bb17-b7d8-47e8-a6a6-056c46360439" containerName="revoke-keys"
Nov 03 11:23:15.129561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129502    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy"
Nov 03 11:23:15.129561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129514    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b38bf7-aad0-494a-9e2a-13dd62b782c7" containerName="istio-proxy"
Nov 03 11:23:15.129561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.129524    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d6bb17-b7d8-47e8-a6a6-056c46360439" containerName="revoke-keys"
Nov 03 11:23:15.135343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.135321    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.139849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.139826    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:23:15.140002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.139851    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:23:15.142841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.142820    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"]
Nov 03 11:23:15.200276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.200245    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.200440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.200311    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.200509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.200482    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4fv\" (UniqueName: \"kubernetes.io/projected/eca9371f-31dd-42c9-bbea-f85a98f71a8b-kube-api-access-gk4fv\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.302870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.302494    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4fv\" (UniqueName: \"kubernetes.io/projected/eca9371f-31dd-42c9-bbea-f85a98f71a8b-kube-api-access-gk4fv\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.302870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.302817    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.303192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.302917    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.309608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.309553    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:15.318760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.310650    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.318760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.316388    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.322992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.322194    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4fv\" (UniqueName: \"kubernetes.io/projected/eca9371f-31dd-42c9-bbea-f85a98f71a8b-kube-api-access-gk4fv\") pod \"maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.461933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.461898    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:15.750083 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:23:15.750055    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca9371f_31dd_42c9_bbea_f85a98f71a8b.slice/crio-8eac46d39a4793d87a5777078e483d6d537221e6a38c509fc3d3794067daa8f9 WatchSource:0}: Error finding container 8eac46d39a4793d87a5777078e483d6d537221e6a38c509fc3d3794067daa8f9: Status 404 returned error can't find the container with id 8eac46d39a4793d87a5777078e483d6d537221e6a38c509fc3d3794067daa8f9
Nov 03 11:23:15.752292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:15.752268    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"]
Nov 03 11:23:16.093064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:16.087785    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:16.736601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:16.721226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:16.793955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:16.793859    5817 generic.go:356] "Generic (PLEG): container finished" podID="eca9371f-31dd-42c9-bbea-f85a98f71a8b" containerID="7af377584495a5eaafad930370863ed77d5875039a326d09a24181904f9cd26a" exitCode=0
Nov 03 11:23:16.794138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:16.794039    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7" event={"ID":"eca9371f-31dd-42c9-bbea-f85a98f71a8b","Type":"ContainerDied","Data":"7af377584495a5eaafad930370863ed77d5875039a326d09a24181904f9cd26a"}
Nov 03 11:23:16.794138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:16.794088    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7" event={"ID":"eca9371f-31dd-42c9-bbea-f85a98f71a8b","Type":"ContainerStarted","Data":"8eac46d39a4793d87a5777078e483d6d537221e6a38c509fc3d3794067daa8f9"}
Nov 03 11:23:17.394458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:17.393380    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:17.991660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:17.986082    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:18.039636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.037899    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:18.057076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.057004    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-openshift-ca\") pod \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") "
Nov 03 11:23:18.057255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.057093    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca9371f-31dd-42c9-bbea-f85a98f71a8b-kube-api-access-gk4fv\" (UniqueName: \"kubernetes.io/projected/eca9371f-31dd-42c9-bbea-f85a98f71a8b-kube-api-access-gk4fv\") pod \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") "
Nov 03 11:23:18.057255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.057233    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-xks-ca\") pod \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\" (UID: \"eca9371f-31dd-42c9-bbea-f85a98f71a8b\") "
Nov 03 11:23:18.057434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.057405    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-openshift-ca" pod "eca9371f-31dd-42c9-bbea-f85a98f71a8b" (UID: "eca9371f-31dd-42c9-bbea-f85a98f71a8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:18.065047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.065016    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:18.072960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.072927    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-xks-ca" pod "eca9371f-31dd-42c9-bbea-f85a98f71a8b" (UID: "eca9371f-31dd-42c9-bbea-f85a98f71a8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:23:18.093536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.093503    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca9371f-31dd-42c9-bbea-f85a98f71a8b-kube-api-access-gk4fv" pod "eca9371f-31dd-42c9-bbea-f85a98f71a8b" (UID: "eca9371f-31dd-42c9-bbea-f85a98f71a8b"). InnerVolumeSpecName "kube-api-access-gk4fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:18.166207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.166173    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9371f-31dd-42c9-bbea-f85a98f71a8b-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:18.166207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.166204    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4fv\" (UniqueName: \"kubernetes.io/projected/eca9371f-31dd-42c9-bbea-f85a98f71a8b-kube-api-access-gk4fv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:18.656605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.655811    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:18.791106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.791062    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:18.816779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.816751    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"
Nov 03 11:23:18.816962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.816781    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7" event={"ID":"eca9371f-31dd-42c9-bbea-f85a98f71a8b","Type":"ContainerDied","Data":"8eac46d39a4793d87a5777078e483d6d537221e6a38c509fc3d3794067daa8f9"}
Nov 03 11:23:18.816962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:18.816878    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eac46d39a4793d87a5777078e483d6d537221e6a38c509fc3d3794067daa8f9"
Nov 03 11:23:19.372770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.372733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:19.783922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.783897    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:23:19.834353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.833517    5817 generic.go:356] "Generic (PLEG): container finished" podID="6f823003-51a6-438f-8270-5fe47685aa9e" containerID="fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6" exitCode=0
Nov 03 11:23:19.834353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.833678    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"
Nov 03 11:23:19.834353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.834229    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" event={"ID":"6f823003-51a6-438f-8270-5fe47685aa9e","Type":"ContainerDied","Data":"fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6"}
Nov 03 11:23:19.834353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.834321    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl" event={"ID":"6f823003-51a6-438f-8270-5fe47685aa9e","Type":"ContainerDied","Data":"ab37fc758981128742e9dec0d2e434b187008f200d9ef18ed497dc0fa1ebce8b"}
Nov 03 11:23:19.834353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.834346    5817 scope.go:122] "RemoveContainer" containerID="fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6"
Nov 03 11:23:19.851376 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.851357    5817 scope.go:122] "RemoveContainer" containerID="59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92"
Nov 03 11:23:19.863210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.863166    5817 scope.go:122] "RemoveContainer" containerID="fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6"
Nov 03 11:23:19.863512 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:19.863492    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6\": container with ID starting with fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6 not found: ID does not exist" containerID="fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6"
Nov 03 11:23:19.863634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.863523    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6"} err="failed to get container status \"fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6\": rpc error: code = NotFound desc = could not find container \"fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6\": container with ID starting with fab4fd64fef636bfa391a46c1132632979e2a19840d32fa2182d1100f71970c6 not found: ID does not exist"
Nov 03 11:23:19.863634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.863546    5817 scope.go:122] "RemoveContainer" containerID="59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92"
Nov 03 11:23:19.863893 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:19.863871    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92\": container with ID starting with 59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92 not found: ID does not exist" containerID="59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92"
Nov 03 11:23:19.863949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.863898    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92"} err="failed to get container status \"59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92\": rpc error: code = NotFound desc = could not find container \"59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92\": container with ID starting with 59711afdbfadde3607db89876fc5807adfa377bb06a82b5800b772b2ea92bc92 not found: ID does not exist"
Nov 03 11:23:19.903115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903086    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-kube-api-access-b49j8\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-kube-api-access-b49j8\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.903279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903138    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-data\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.903365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903345    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-envoy\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.903445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903423    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-data" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.903608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903559    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f823003-51a6-438f-8270-5fe47685aa9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f823003-51a6-438f-8270-5fe47685aa9e-istiod-ca-cert\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.903721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903658    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-credential-socket\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.903809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903777    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-socket\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.903891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903861    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f823003-51a6-438f-8270-5fe47685aa9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f823003-51a6-438f-8270-5fe47685aa9e-istio-podinfo\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.903998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903896    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-certs\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.903998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903962    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-istio-token\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-istio-token\") pod \"6f823003-51a6-438f-8270-5fe47685aa9e\" (UID: \"6f823003-51a6-438f-8270-5fe47685aa9e\") "
Nov 03 11:23:19.904196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.903929    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f823003-51a6-438f-8270-5fe47685aa9e-istiod-ca-cert" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:19.904432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.904359    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-certs" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.904432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.904391    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-socket" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.904699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.904470    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-credential-socket" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.904988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.904863    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f823003-51a6-438f-8270-5fe47685aa9e-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.904988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.904887    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.904988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.904919    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.905262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.905036    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.905262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.905090    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.906658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.906635    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-istio-token" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:19.907258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.907236    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f823003-51a6-438f-8270-5fe47685aa9e-istio-podinfo" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:23:19.907341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.907238    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-kube-api-access-b49j8" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "kube-api-access-b49j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:19.907635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.907616    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-envoy" pod "6f823003-51a6-438f-8270-5fe47685aa9e" (UID: "6f823003-51a6-438f-8270-5fe47685aa9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.988026 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:19.987789    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:20.015729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:20.008267    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f823003-51a6-438f-8270-5fe47685aa9e-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:20.015729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:20.008303    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:20.015729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:20.008322    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b49j8\" (UniqueName: \"kubernetes.io/projected/6f823003-51a6-438f-8270-5fe47685aa9e-kube-api-access-b49j8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:20.015729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:20.008337    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f823003-51a6-438f-8270-5fe47685aa9e-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:20.168214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:20.168185    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"]
Nov 03 11:23:20.176504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:20.176448    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-68aa46-openshift-default-696d457fcf-8fcrl"]
Nov 03 11:23:20.205296 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:20.205265    5817 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f823003_51a6_438f_8270_5fe47685aa9e.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:23:20.740689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:20.739123    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:21.445659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:21.445620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:21.905420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:21.905342    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" path="/var/lib/kubelet/pods/6f823003-51a6-438f-8270-5fe47685aa9e/volumes"
Nov 03 11:23:22.076596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:22.075466    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:22.416957 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:22.416909    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:23:22.687661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:22.682557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:23.278597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:23.272044    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:23.871675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:23.869470    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:24.490801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:24.490762    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:25.204775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:25.204726    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:25.285065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:25.285036    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"]
Nov 03 11:23:25.298365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:25.298337    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"]
Nov 03 11:23:25.404628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:25.404596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:25.576662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:25.574889    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:26.167880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:26.157428    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:26.744243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:26.744166    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:27.386783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:27.386733    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:28.077619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:28.076161    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:28.654647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:28.654615    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:29.291029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:29.290429    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:30.083602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.078617    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:30.285608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.285543    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts" podUID="6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" containerName="payload-processing" containerID="cri-o://f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903" gracePeriod=25
Nov 03 11:23:30.299132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.299090    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" podUID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" containerName="payload-pre-processing" containerID="cri-o://b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769" gracePeriod=25
Nov 03 11:23:30.430975 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:23:30.430948    5817 logging.go:55] [core] [Channel #15554 SubChannel #15556]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.77:50052", ServerName: "10.133.1.77:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.77:50052: connect: connection refused"
Nov 03 11:23:30.431960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.431923    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" podUID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:23:30.600935 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.600906    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:23:30.644740 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.644709    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-kube-api-access-t9tqj\" (UniqueName: \"kubernetes.io/projected/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-kube-api-access-t9tqj\") pod \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\" (UID: \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\") "
Nov 03 11:23:30.644988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.644970    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-plugins-config-volume\") pod \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\" (UID: \"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82\") "
Nov 03 11:23:30.645428 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.645398    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-plugins-config-volume" pod "6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" (UID: "6d45b3c2-9964-4e3f-85c7-830f2bbf1b82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:30.645692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.645671    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:30.647495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.647465    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-kube-api-access-t9tqj" pod "6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" (UID: "6d45b3c2-9964-4e3f-85c7-830f2bbf1b82"). InnerVolumeSpecName "kube-api-access-t9tqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:30.746830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.746801    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9tqj\" (UniqueName: \"kubernetes.io/projected/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82-kube-api-access-t9tqj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:30.795391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:30.795172    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:31.046962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.046930    5817 generic.go:356] "Generic (PLEG): container finished" podID="6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" containerID="f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903" exitCode=0
Nov 03 11:23:31.047125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.046978    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts" event={"ID":"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82","Type":"ContainerDied","Data":"f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903"}
Nov 03 11:23:31.047125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.047019    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts" event={"ID":"6d45b3c2-9964-4e3f-85c7-830f2bbf1b82","Type":"ContainerDied","Data":"507d1815c8f1853a47e5c1bbd7940cd039b8bd28a1833c828322ebce1f52334e"}
Nov 03 11:23:31.047125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.047042    5817 scope.go:122] "RemoveContainer" containerID="f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903"
Nov 03 11:23:31.047125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.046980    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"
Nov 03 11:23:31.060301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.060279    5817 scope.go:122] "RemoveContainer" containerID="f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903"
Nov 03 11:23:31.060551 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:31.060523    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903\": container with ID starting with f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903 not found: ID does not exist" containerID="f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903"
Nov 03 11:23:31.060759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.060564    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903"} err="failed to get container status \"f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903\": rpc error: code = NotFound desc = could not find container \"f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903\": container with ID starting with f5ecd0c427c157b5b45eb605c14a67771cfdc176bbc4875695249301ff1ad903 not found: ID does not exist"
Nov 03 11:23:31.073321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.073287    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"]
Nov 03 11:23:31.078233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.078199    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts"]
Nov 03 11:23:31.320393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.320304    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:23:31.433281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.433239    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c0a748-76c9674b77-x44ts" podUID="6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.77:50052\" within 1s: context deadline exceeded"
Nov 03 11:23:31.433886 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:23:31.433861    5817 logging.go:55] [core] [Channel #15554 SubChannel #15556]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.77:50052", ServerName: "10.133.1.77:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.77:50052: operation was canceled"
Nov 03 11:23:31.472039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.468158    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:31.904169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:31.904137    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" path="/var/lib/kubelet/pods/6d45b3c2-9964-4e3f-85c7-830f2bbf1b82/volumes"
Nov 03 11:23:32.115942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:32.115909    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:32.331606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:32.331242    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:32.479596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:32.478960    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:32.620727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:32.620629    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"]
Nov 03 11:23:32.628411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:32.628381    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c0a748-95rd7"]
Nov 03 11:23:32.774599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:32.772268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:33.073018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:33.072985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:33.502614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:33.500431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:33.904375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:33.904289    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca9371f-31dd-42c9-bbea-f85a98f71a8b" path="/var/lib/kubelet/pods/eca9371f-31dd-42c9-bbea-f85a98f71a8b/volumes"
Nov 03 11:23:34.197451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:34.187156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:34.889595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:34.880280    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:35.019847 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:35.019813    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:35.303183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:35.303151    5817 scope.go:122] "RemoveContainer" containerID="d843dd7ea15d6547ea8ffece88f8cfbad66de9b08150c71639b7c7985fbd0e98"
Nov 03 11:23:35.463596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:35.459309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:36.176859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:36.176824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:36.806737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:36.804900    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:37.377741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:37.377691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:38.063020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:38.057448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:38.081437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:38.081394    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"]
Nov 03 11:23:38.086840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:38.086814    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb846996-m42sb"]
Nov 03 11:23:38.767646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:38.767607    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:39.308201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:39.308165    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:39.464594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:39.461215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:39.575003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:39.574922    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"]
Nov 03 11:23:39.575467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:39.575439    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy" containerID="cri-o://2f6b37cf17f6b81d5f8bd5b593193b83929f1433c86b6a16446e1f4f00115985" gracePeriod=30
Nov 03 11:23:39.609848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:39.609021    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"]
Nov 03 11:23:39.609848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:39.609252    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy" containerID="cri-o://3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191" gracePeriod=30
Nov 03 11:23:39.641279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:39.641249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:39.926678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:39.926597    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c59a54-c4d5-4e02-a940-ad458f72f559" path="/var/lib/kubelet/pods/65c59a54-c4d5-4e02-a940-ad458f72f559/volumes"
Nov 03 11:23:40.300229 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:40.297307    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:40.461596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:40.453991    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" podUID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:23:42.270818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:42.270779    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:42.866815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:42.862112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:43.568795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:43.568758    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:44.484765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:44.484718    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:44.860623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:44.858186    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:44.911993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:44.911958    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:23:45.031618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.031463    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-token\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-token\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.031618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.031541    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5308b6-3a71-4bca-9b07-fe824d145610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5308b6-3a71-4bca-9b07-fe824d145610-istiod-ca-cert\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.031865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.031672    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-podinfo\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.031865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.031730    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-credential-socket\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.031865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.031762    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-envoy\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.031865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.031805    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-kube-api-access-bflzv\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-kube-api-access-bflzv\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.032110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.031882    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-certs\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.032110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.031918    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-data\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.032218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.032137    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-socket\") pod \"8a5308b6-3a71-4bca-9b07-fe824d145610\" (UID: \"8a5308b6-3a71-4bca-9b07-fe824d145610\") "
Nov 03 11:23:45.033409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.033013    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5308b6-3a71-4bca-9b07-fe824d145610-istiod-ca-cert" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:45.033409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.033233    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-socket" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.033691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.033433    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-data" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.033691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.033493    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-credential-socket" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.033691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.033589    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-certs" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.036933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.036909    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-podinfo" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:23:45.037175 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.037157    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-kube-api-access-bflzv" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "kube-api-access-bflzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:45.037349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.037333    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-envoy" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.037780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.037750    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-token" pod "8a5308b6-3a71-4bca-9b07-fe824d145610" (UID: "8a5308b6-3a71-4bca-9b07-fe824d145610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:45.134838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134808    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.134838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134835    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5308b6-3a71-4bca-9b07-fe824d145610-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.135050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134847    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.135050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134857    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.135050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134865    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.135050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134875    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bflzv\" (UniqueName: \"kubernetes.io/projected/8a5308b6-3a71-4bca-9b07-fe824d145610-kube-api-access-bflzv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.135050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134885    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.135050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134893    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.135050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.134901    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5308b6-3a71-4bca-9b07-fe824d145610-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.279589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.279542    5817 generic.go:356] "Generic (PLEG): container finished" podID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerID="2f6b37cf17f6b81d5f8bd5b593193b83929f1433c86b6a16446e1f4f00115985" exitCode=0
Nov 03 11:23:45.280033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.280007    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" event={"ID":"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a","Type":"ContainerDied","Data":"2f6b37cf17f6b81d5f8bd5b593193b83929f1433c86b6a16446e1f4f00115985"}
Nov 03 11:23:45.283025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.282968    5817 generic.go:356] "Generic (PLEG): container finished" podID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerID="3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191" exitCode=0
Nov 03 11:23:45.283130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.283044    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" event={"ID":"8a5308b6-3a71-4bca-9b07-fe824d145610","Type":"ContainerDied","Data":"3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191"}
Nov 03 11:23:45.283130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.283086    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd" event={"ID":"8a5308b6-3a71-4bca-9b07-fe824d145610","Type":"ContainerDied","Data":"f53fac2db11aafa275eba72b4ee44c91d75b980a52f493b51147f8cd677023a3"}
Nov 03 11:23:45.283130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.283109    5817 scope.go:122] "RemoveContainer" containerID="3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191"
Nov 03 11:23:45.283367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.283226    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"
Nov 03 11:23:45.288058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.287922    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:45.297553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.297257    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:23:45.302658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.302637    5817 scope.go:122] "RemoveContainer" containerID="204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec"
Nov 03 11:23:45.318754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.318735    5817 scope.go:122] "RemoveContainer" containerID="3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191"
Nov 03 11:23:45.319046 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:45.319024    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191\": container with ID starting with 3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191 not found: ID does not exist" containerID="3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191"
Nov 03 11:23:45.319158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.319052    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191"} err="failed to get container status \"3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191\": rpc error: code = NotFound desc = could not find container \"3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191\": container with ID starting with 3542bdad9e90513dee94c670abcf2e80ae89938663fc4b16cdd02860893ee191 not found: ID does not exist"
Nov 03 11:23:45.319158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.319069    5817 scope.go:122] "RemoveContainer" containerID="204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec"
Nov 03 11:23:45.319345 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:45.319324    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec\": container with ID starting with 204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec not found: ID does not exist" containerID="204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec"
Nov 03 11:23:45.319409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.319356    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec"} err="failed to get container status \"204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec\": rpc error: code = NotFound desc = could not find container \"204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec\": container with ID starting with 204172aba32cd29d407dac620210c9a90d325c8c5b6f8650c45f977c1616f6ec not found: ID does not exist"
Nov 03 11:23:45.358103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.358034    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"]
Nov 03 11:23:45.364335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.364308    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-stlsd"]
Nov 03 11:23:45.454696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.448464    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-credential-socket\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.454696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.448667    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-kube-api-access-l8f6f\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-kube-api-access-l8f6f\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.454696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.448799    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-podinfo\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.454696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.448843    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-certs\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.454696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.448904    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-envoy\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.454696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.448940    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-socket\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.454696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.449009    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-token\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-token\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.501712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.472729    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-data\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.501712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.472905    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istiod-ca-cert\") pod \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\" (UID: \"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a\") "
Nov 03 11:23:45.502681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.502429    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istiod-ca-cert" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:45.503177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.502874    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-certs" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.541727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.530942    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:45.552595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.534860    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-credential-socket" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.552595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.536513    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-podinfo" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:23:45.552595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.547977    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.552595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.548006    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.552595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.548023    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.552595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.548039    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.567631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.556704    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-data" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.567631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.557649    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-socket" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.677697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.653252    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.677697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.653292    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.681534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.681494    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-envoy" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.681926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.681605    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-kube-api-access-l8f6f" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "kube-api-access-l8f6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:45.698544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.683845    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-token" pod "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" (UID: "7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:45.754459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.754428    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8f6f\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-kube-api-access-l8f6f\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.754459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.754457    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.754679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.754472    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.904525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:45.904445    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" path="/var/lib/kubelet/pods/8a5308b6-3a71-4bca-9b07-fe824d145610/volumes"
Nov 03 11:23:46.267139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.266611    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:46.330323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.330292    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"
Nov 03 11:23:46.330507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.330289    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk" event={"ID":"7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a","Type":"ContainerDied","Data":"3f17cd783f3a9d8e9870cb6c5073cdf261a73ec62e18f7fa829f80e73fe6488f"}
Nov 03 11:23:46.330507 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.330482    5817 scope.go:122] "RemoveContainer" containerID="2f6b37cf17f6b81d5f8bd5b593193b83929f1433c86b6a16446e1f4f00115985"
Nov 03 11:23:46.354395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.354362    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"]
Nov 03 11:23:46.362896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.362874    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-s64xk"]
Nov 03 11:23:46.532781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.532695    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"]
Nov 03 11:23:46.533534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533513    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533536    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533561    5817 cpu_manager.go:414] "containerMap: removing container" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533585    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533603    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" containerName="payload-processing"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533610    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" containerName="payload-processing"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533644    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533651    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533663    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy"
Nov 03 11:23:46.533679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533670    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533708    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533715    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533739    5817 cpu_manager.go:414] "containerMap: removing container" podUID="eca9371f-31dd-42c9-bbea-f85a98f71a8b" containerName="revoke-keys"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533746    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca9371f-31dd-42c9-bbea-f85a98f71a8b" containerName="revoke-keys"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533758    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533765    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533966    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca9371f-31dd-42c9-bbea-f85a98f71a8b" containerName="revoke-keys"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533979    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.533991    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.534003    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea678e2-cfc3-4ff5-8889-601be2dd2909" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.534011    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d45b3c2-9964-4e3f-85c7-830f2bbf1b82" containerName="payload-processing"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.534025    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" containerName="istio-proxy"
Nov 03 11:23:46.534160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.534033    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f823003-51a6-438f-8270-5fe47685aa9e" containerName="istio-proxy"
Nov 03 11:23:46.545494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.545468    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:46.549039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.549010    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"]
Nov 03 11:23:46.552147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.552102    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-049d3b4c-dockercfg-95lnk\""
Nov 03 11:23:46.552901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.552880    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-049d3b4c\""
Nov 03 11:23:46.554904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.554281    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:46.565065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.565029    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"]
Nov 03 11:23:46.568663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.568639    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5308b6-3a71-4bca-9b07-fe824d145610" containerName="istio-proxy"
Nov 03 11:23:46.576446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.576425    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:46.600895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.600868    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"]
Nov 03 11:23:46.681595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.678995    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f7c03-e465-4d5c-80e8-ae88da2976a9-plugins-config-volume\") pod \"payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh\" (UID: \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\") " pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:46.681595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.679173    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d1b8bb-edae-4b26-9efc-0418a566a48d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx\" (UID: \"91d1b8bb-edae-4b26-9efc-0418a566a48d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:46.681595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.679222    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8mh\" (UniqueName: \"kubernetes.io/projected/c83f7c03-e465-4d5c-80e8-ae88da2976a9-kube-api-access-hq8mh\") pod \"payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh\" (UID: \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\") " pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:46.681595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.679276    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtsv\" (UniqueName: \"kubernetes.io/projected/91d1b8bb-edae-4b26-9efc-0418a566a48d-kube-api-access-ljtsv\") pod \"payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx\" (UID: \"91d1b8bb-edae-4b26-9efc-0418a566a48d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:46.718621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.716883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:46.790729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.784642    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f7c03-e465-4d5c-80e8-ae88da2976a9-plugins-config-volume\") pod \"payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh\" (UID: \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\") " pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:46.790729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.784822    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d1b8bb-edae-4b26-9efc-0418a566a48d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx\" (UID: \"91d1b8bb-edae-4b26-9efc-0418a566a48d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:46.790729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.784875    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8mh\" (UniqueName: \"kubernetes.io/projected/c83f7c03-e465-4d5c-80e8-ae88da2976a9-kube-api-access-hq8mh\") pod \"payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh\" (UID: \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\") " pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:46.790729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.784935    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtsv\" (UniqueName: \"kubernetes.io/projected/91d1b8bb-edae-4b26-9efc-0418a566a48d-kube-api-access-ljtsv\") pod \"payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx\" (UID: \"91d1b8bb-edae-4b26-9efc-0418a566a48d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:46.790729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.786322    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f7c03-e465-4d5c-80e8-ae88da2976a9-plugins-config-volume\") pod \"payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh\" (UID: \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\") " pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:46.790729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.787065    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d1b8bb-edae-4b26-9efc-0418a566a48d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx\" (UID: \"91d1b8bb-edae-4b26-9efc-0418a566a48d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:46.880606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.876293    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtsv\" (UniqueName: \"kubernetes.io/projected/91d1b8bb-edae-4b26-9efc-0418a566a48d-kube-api-access-ljtsv\") pod \"payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx\" (UID: \"91d1b8bb-edae-4b26-9efc-0418a566a48d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:46.883951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.883922    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8mh\" (UniqueName: \"kubernetes.io/projected/c83f7c03-e465-4d5c-80e8-ae88da2976a9-kube-api-access-hq8mh\") pod \"payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh\" (UID: \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\") " pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:46.887536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.886845    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:46.898838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:46.898776    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:47.249095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:47.249066    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"]
Nov 03 11:23:47.249498 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:23:47.249242    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83f7c03_e465_4d5c_80e8_ae88da2976a9.slice/crio-31a73c320733ce11555a9f98b67c5f0b5201718a119c26077257916b8115de0f WatchSource:0}: Error finding container 31a73c320733ce11555a9f98b67c5f0b5201718a119c26077257916b8115de0f: Status 404 returned error can't find the container with id 31a73c320733ce11555a9f98b67c5f0b5201718a119c26077257916b8115de0f
Nov 03 11:23:47.260827 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:23:47.260788    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d1b8bb_edae_4b26_9efc_0418a566a48d.slice/crio-a9401671cd5ecbd94cd3b05f8c1a734cf6d942a1348afc934aa09d80362c0dbd WatchSource:0}: Error finding container a9401671cd5ecbd94cd3b05f8c1a734cf6d942a1348afc934aa09d80362c0dbd: Status 404 returned error can't find the container with id a9401671cd5ecbd94cd3b05f8c1a734cf6d942a1348afc934aa09d80362c0dbd
Nov 03 11:23:47.260827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:47.260807    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"]
Nov 03 11:23:47.346324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:47.346293    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx" event={"ID":"91d1b8bb-edae-4b26-9efc-0418a566a48d","Type":"ContainerStarted","Data":"a9401671cd5ecbd94cd3b05f8c1a734cf6d942a1348afc934aa09d80362c0dbd"}
Nov 03 11:23:47.349692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:47.349663    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh" event={"ID":"c83f7c03-e465-4d5c-80e8-ae88da2976a9","Type":"ContainerStarted","Data":"31a73c320733ce11555a9f98b67c5f0b5201718a119c26077257916b8115de0f"}
Nov 03 11:23:47.349815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:47.349721    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:47.390599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:47.385859    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh" podStartSLOduration=1.3858356760000001 podStartE2EDuration="1.385835676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:23:47.372120844 +0000 UTC m=+3257.577702896" watchObservedRunningTime="2026-11-03 11:23:47.385835676 +0000 UTC m=+3257.591417731"
Nov 03 11:23:47.418209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:47.418110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:47.906710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:47.906672    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a" path="/var/lib/kubelet/pods/7dcbfc0b-0e4b-4a69-bf0c-3f0065308b4a/volumes"
Nov 03 11:23:48.166692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:48.165362    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:48.364165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:48.364129    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx" event={"ID":"91d1b8bb-edae-4b26-9efc-0418a566a48d","Type":"ContainerStarted","Data":"f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517"}
Nov 03 11:23:48.364734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:48.364233    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:48.366316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:48.366286    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh" event={"ID":"c83f7c03-e465-4d5c-80e8-ae88da2976a9","Type":"ContainerStarted","Data":"7a67c8865c4006d2ec2c10fca51bdba688b9672d5d249ec736b7d522a56fde51"}
Nov 03 11:23:48.393883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:48.393833    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx" podStartSLOduration=2.39382069 podStartE2EDuration="2.39382069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:23:48.388064728 +0000 UTC m=+3258.593646780" watchObservedRunningTime="2026-11-03 11:23:48.39382069 +0000 UTC m=+3258.599402741"
Nov 03 11:23:48.869656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:48.866656    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:49.026712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:49.026677    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"]
Nov 03 11:23:49.031174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:49.031151    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857df2ab-l5xxz"]
Nov 03 11:23:49.679489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:49.679454    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:49.908857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:49.908771    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c96d741-f606-49f3-8b41-3e85be4f1558" path="/var/lib/kubelet/pods/9c96d741-f606-49f3-8b41-3e85be4f1558/volumes"
Nov 03 11:23:50.415601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:50.412166    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:50.431286 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:23:50.431252    5817 logging.go:55] [core] [Channel #15730 SubChannel #15731]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.76:50052", ServerName: "10.133.1.76:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.76:50052: connect: connection refused"
Nov 03 11:23:50.637099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:50.637069    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:23:50.673476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:50.673444    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0f4d83-bc3f-4c99-9173-1efcb60086de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0f4d83-bc3f-4c99-9173-1efcb60086de-plugins-config-volume\") pod \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\" (UID: \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\") "
Nov 03 11:23:50.674034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:50.673727    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0f4d83-bc3f-4c99-9173-1efcb60086de-kube-api-access-775cm\" (UniqueName: \"kubernetes.io/projected/ad0f4d83-bc3f-4c99-9173-1efcb60086de-kube-api-access-775cm\") pod \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\" (UID: \"ad0f4d83-bc3f-4c99-9173-1efcb60086de\") "
Nov 03 11:23:50.674541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:50.674518    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0f4d83-bc3f-4c99-9173-1efcb60086de-plugins-config-volume" pod "ad0f4d83-bc3f-4c99-9173-1efcb60086de" (UID: "ad0f4d83-bc3f-4c99-9173-1efcb60086de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:50.676059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:50.676033    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0f4d83-bc3f-4c99-9173-1efcb60086de-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:50.698814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:50.697675    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0f4d83-bc3f-4c99-9173-1efcb60086de-kube-api-access-775cm" pod "ad0f4d83-bc3f-4c99-9173-1efcb60086de" (UID: "ad0f4d83-bc3f-4c99-9173-1efcb60086de"). InnerVolumeSpecName "kube-api-access-775cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:50.778250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:50.778213    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-775cm\" (UniqueName: \"kubernetes.io/projected/ad0f4d83-bc3f-4c99-9173-1efcb60086de-kube-api-access-775cm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:51.038596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.036152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:51.431209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.431165    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" podUID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.76:50052\" within 1s: context deadline exceeded"
Nov 03 11:23:51.431396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.431260    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:23:51.432842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.432816    5817 generic.go:356] "Generic (PLEG): container finished" podID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" containerID="b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769" exitCode=0
Nov 03 11:23:51.432982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.432857    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"
Nov 03 11:23:51.432982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.432904    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" event={"ID":"ad0f4d83-bc3f-4c99-9173-1efcb60086de","Type":"ContainerDied","Data":"b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769"}
Nov 03 11:23:51.432982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.432959    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx" event={"ID":"ad0f4d83-bc3f-4c99-9173-1efcb60086de","Type":"ContainerDied","Data":"85bcf4259349dc98f2d14955de9a9923a1aa917a99bfdf3cd44010267551feab"}
Nov 03 11:23:51.433147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.432993    5817 scope.go:122] "RemoveContainer" containerID="b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769"
Nov 03 11:23:51.445876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.445852    5817 scope.go:122] "RemoveContainer" containerID="b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769"
Nov 03 11:23:51.446258 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:51.446230    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769\": container with ID starting with b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769 not found: ID does not exist" containerID="b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769"
Nov 03 11:23:51.446369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.446263    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769"} err="failed to get container status \"b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769\": rpc error: code = NotFound desc = could not find container \"b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769\": container with ID starting with b5bfbbbb60d99d2ac939aa31edde8f8acbbb632ab50a6e7e02d5c1c69dc0f769 not found: ID does not exist"
Nov 03 11:23:51.463086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.463063    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"]
Nov 03 11:23:51.471258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.471232    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c0a748-6f744bc5f9-f6xmx"]
Nov 03 11:23:51.639991 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.639835    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:51.907066 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:51.906982    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" path="/var/lib/kubelet/pods/ad0f4d83-bc3f-4c99-9173-1efcb60086de/volumes"
Nov 03 11:23:52.366607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:52.363064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:52.433792 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:23:52.433754    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:23:53.067834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:53.067782    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:53.682292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:53.682258    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:54.370416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:54.370313    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:54.529965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:54.529925    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:55.060445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:55.060412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:55.680318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:55.680274    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:56.365350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:56.365247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:57.078599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:57.072103    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:57.760286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:57.760249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:58.561592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:58.561533    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:59.261419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:59.258847    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:23:59.377354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:59.377327    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:23:59.377540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:59.377394    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:23:59.889202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:23:59.889110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:00.569257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:00.569224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:01.241081 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:01.240807    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:01.839219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:01.839180    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:02.569122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:02.569083    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:03.204207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:03.203332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:03.805594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:03.798814    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:04.672196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:04.672156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:04.817596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:04.816710    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:05.544599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:05.521093    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:06.176353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:06.176313    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:06.970596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:06.968379    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:07.669600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:07.663664    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:08.288225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:08.288191    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:08.500456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:08.500416    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:09.113404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:09.108876    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:09.890192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:09.890156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:10.589883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:10.589843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:11.181594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:11.181240    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:11.874253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:11.867367    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:12.030591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:12.030369    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:12.480600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:12.471986    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:13.108342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:13.108308    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:13.995557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:13.995523    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:14.770231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:14.770188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:15.402309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:15.402272    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:16.170126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:16.170091    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:16.733504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:16.733468    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:16.929642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:16.926012    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:17.149447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:17.149343    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:17.974974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:17.974940    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:18.174068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:18.174035    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:18.844917 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:18.844888    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:19.019224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.019189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:19.729621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.728157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:19.737982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.737953    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"]
Nov 03 11:24:19.739924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.739700    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" containerName="payload-pre-processing"
Nov 03 11:24:19.740149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.740132    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" containerName="payload-pre-processing"
Nov 03 11:24:19.740594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.740559    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0f4d83-bc3f-4c99-9173-1efcb60086de" containerName="payload-pre-processing"
Nov 03 11:24:19.745444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.745424    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"]
Nov 03 11:24:19.745688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.745673    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.751452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.751432    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:24:19.752244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.752227    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:24:19.789074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.789035    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnf6f\" (UniqueName: \"kubernetes.io/projected/09a31bce-9447-4ee9-b1f5-bb74ac93834e-kube-api-access-pnf6f\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.789227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.789202    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.789297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.789251    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.891186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.891154    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnf6f\" (UniqueName: \"kubernetes.io/projected/09a31bce-9447-4ee9-b1f5-bb74ac93834e-kube-api-access-pnf6f\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.891342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.891278    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.891342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.891323    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.892098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.892071    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.894072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.894044    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:19.903115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:19.903092    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnf6f\" (UniqueName: \"kubernetes.io/projected/09a31bce-9447-4ee9-b1f5-bb74ac93834e-kube-api-access-pnf6f\") pod \"maas-api-revoke-sub-f006aed1-dxntd\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:20.073267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:20.073192    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:20.366644 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:24:20.366612    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a31bce_9447_4ee9_b1f5_bb74ac93834e.slice/crio-db099a666f77bbe7b84eff96076e9476eda3650d9994ae192be8d4f29f32f29b WatchSource:0}: Error finding container db099a666f77bbe7b84eff96076e9476eda3650d9994ae192be8d4f29f32f29b: Status 404 returned error can't find the container with id db099a666f77bbe7b84eff96076e9476eda3650d9994ae192be8d4f29f32f29b
Nov 03 11:24:20.369042 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:20.369019    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"]
Nov 03 11:24:20.595596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:20.595343    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:20.784737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:20.784705    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd" event={"ID":"09a31bce-9447-4ee9-b1f5-bb74ac93834e","Type":"ContainerStarted","Data":"db099a666f77bbe7b84eff96076e9476eda3650d9994ae192be8d4f29f32f29b"}
Nov 03 11:24:21.273597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:21.272630    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:21.809039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:21.808999    5817 generic.go:356] "Generic (PLEG): container finished" podID="09a31bce-9447-4ee9-b1f5-bb74ac93834e" containerID="3d3f0eca28260aebd96b6b66e96442b98c69e6a5ca463fd84527d401cf3967a9" exitCode=0
Nov 03 11:24:21.809252 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:21.809065    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd" event={"ID":"09a31bce-9447-4ee9-b1f5-bb74ac93834e","Type":"ContainerDied","Data":"3d3f0eca28260aebd96b6b66e96442b98c69e6a5ca463fd84527d401cf3967a9"}
Nov 03 11:24:21.973649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:21.972972    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:22.415399 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:24:22.415367    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:24:22.784564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:22.784527    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:23.024805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.024777    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:23.046764 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.046689    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a31bce-9447-4ee9-b1f5-bb74ac93834e-kube-api-access-pnf6f\" (UniqueName: \"kubernetes.io/projected/09a31bce-9447-4ee9-b1f5-bb74ac93834e-kube-api-access-pnf6f\") pod \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") "
Nov 03 11:24:23.046978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.046846    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-openshift-ca\") pod \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") "
Nov 03 11:24:23.046978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.046907    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-xks-ca\") pod \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\" (UID: \"09a31bce-9447-4ee9-b1f5-bb74ac93834e\") "
Nov 03 11:24:23.047483 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.047459    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-openshift-ca" pod "09a31bce-9447-4ee9-b1f5-bb74ac93834e" (UID: "09a31bce-9447-4ee9-b1f5-bb74ac93834e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:24:23.047765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.047743    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:24:23.049791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.049758    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-xks-ca" pod "09a31bce-9447-4ee9-b1f5-bb74ac93834e" (UID: "09a31bce-9447-4ee9-b1f5-bb74ac93834e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:24:23.049905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.049817    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a31bce-9447-4ee9-b1f5-bb74ac93834e-kube-api-access-pnf6f" pod "09a31bce-9447-4ee9-b1f5-bb74ac93834e" (UID: "09a31bce-9447-4ee9-b1f5-bb74ac93834e"). InnerVolumeSpecName "kube-api-access-pnf6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:24:23.152486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.149941    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a31bce-9447-4ee9-b1f5-bb74ac93834e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:24:23.152486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.149975    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnf6f\" (UniqueName: \"kubernetes.io/projected/09a31bce-9447-4ee9-b1f5-bb74ac93834e-kube-api-access-pnf6f\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:24:23.494593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.494286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:23.850137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.850054    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"
Nov 03 11:24:23.850137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.850084    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd" event={"ID":"09a31bce-9447-4ee9-b1f5-bb74ac93834e","Type":"ContainerDied","Data":"db099a666f77bbe7b84eff96076e9476eda3650d9994ae192be8d4f29f32f29b"}
Nov 03 11:24:23.850137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:23.850128    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db099a666f77bbe7b84eff96076e9476eda3650d9994ae192be8d4f29f32f29b"
Nov 03 11:24:24.142644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:24.139156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:24.785892 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:24.785855    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:25.402478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:25.402443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:26.000293 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:26.000253    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:24:26.074715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:26.074631    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:27.080147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:27.080111    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:27.921760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:27.921682    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:28.736526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:28.736491    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:29.588707 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:29.588665    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:30.149232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:30.149195    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:30.379604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:30.379248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:31.081278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:31.081213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:31.254055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:31.254023    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:31.919349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:31.919309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:32.087207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:32.087168    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:32.520096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:32.520054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:32.966632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:32.966594    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:33.694060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:33.694026    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:34.457607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:34.456136    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:35.249348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:35.249123    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:35.381165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:35.381131    5817 scope.go:122] "RemoveContainer" containerID="3b8d22e936e90b0c0e8e3151c13894d4bed0a64f73832613d11be190502e5bf6"
Nov 03 11:24:35.986616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:35.982227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:36.417528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:36.417428    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:36.673360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:36.673267    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:37.518991 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:37.518914    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:38.193665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:38.193626    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:38.917835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:38.917795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:39.086126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:39.086093    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:39.272625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:39.272594    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:39.429990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:39.429942    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:39.822670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:39.822630    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:40.789785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:40.789731    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:41.676377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:41.676303    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:42.366451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:42.366417    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:43.152602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:43.151513    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:43.802671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:43.802560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:44.656445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:44.656411    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:45.568597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:45.565199    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:45.713252 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:45.713216    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:46.268804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:46.268763    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:46.868058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:46.868020    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:47.478152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:47.478119    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:48.668138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:48.668065    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:49.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:49.049387    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:24:49.049597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:49.049458    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:24:49.423650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:49.422649    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:24:49.687713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:49.687675    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:24:49.731479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:49.731444    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:50.030758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:50.030660    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 03 11:24:50.033685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:50.032790    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:24:50.033685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:50.033123    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 11:24:50.152598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:50.112675    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:24:50.158754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:50.158710    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:24:50.365653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:50.361446    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:51.110355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:51.110316    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:51.888565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:51.888528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:52.419739 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:24:52.419706    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:24:52.612593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:52.605786    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:53.428591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:53.428542    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:54.072882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:54.072839    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:54.574649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:54.574596    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:54.844632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:54.844521    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:55.586997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:55.586964    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:56.299384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:56.299349    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:56.968295 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:56.968261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:57.663063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:57.663030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:58.342073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:58.342033    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:59.004598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:58.999197    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:24:59.686077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:24:59.686047    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:00.422459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:00.422428    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:01.190108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:01.179252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:01.810634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:01.809553    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:02.494594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:02.488443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:03.350603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:03.347713    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:03.957271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:03.957233    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:04.579880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:04.579831    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:05.051373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:05.051339    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:05.193954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:05.193921    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:05.358243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:05.358145    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:05.566235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:05.566119    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:06.072597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:06.069641    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:06.811613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:06.810431    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:07.474606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:07.473054    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:08.081162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:08.081126    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:08.550275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:08.550239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:09.013731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:09.013696    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:09.567610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:09.567210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:10.230798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:10.229683    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:10.971602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:10.971540    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:11.534096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:11.534064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:11.738348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:11.738289    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:11.880003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:11.879918    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:12.383150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:12.383110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:13.244401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:13.244349    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:13.976500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:13.976466    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:14.670011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:14.669976    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:14.845640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:14.836715    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:15.537349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:15.537310    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:16.248211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:16.248167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:17.162132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:17.162098    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:17.892916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:17.892881    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:18.585760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:18.584188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:18.886897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:18.886814    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:19.351353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:19.351310    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:19.863994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:19.863952    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:25:20.037584 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:20.037528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:20.944152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:20.944121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:22.169290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:22.169256    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:22.420402 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:25:22.420316    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:25:22.878600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:22.877112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:23.553651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.548690    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:23.859669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.859595    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"]
Nov 03 11:25:23.860371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.860357    5817 cpu_manager.go:414] "containerMap: removing container" podUID="09a31bce-9447-4ee9-b1f5-bb74ac93834e" containerName="revoke-keys"
Nov 03 11:25:23.860418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.860373    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a31bce-9447-4ee9-b1f5-bb74ac93834e" containerName="revoke-keys"
Nov 03 11:25:23.860521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.860513    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a31bce-9447-4ee9-b1f5-bb74ac93834e" containerName="revoke-keys"
Nov 03 11:25:23.864779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.864759    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:23.869225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.869203    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:25:23.869404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.869289    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:25:23.874779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.874757    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"]
Nov 03 11:25:23.929179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.929149    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzmh\" (UniqueName: \"kubernetes.io/projected/ee5ba865-e06b-42d3-ba05-fc5093f54f32-kube-api-access-smzmh\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:23.929381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.929191    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:23.929381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:23.929303    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:24.030472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.030438    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzmh\" (UniqueName: \"kubernetes.io/projected/ee5ba865-e06b-42d3-ba05-fc5093f54f32-kube-api-access-smzmh\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:24.030699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.030485    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:24.030699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.030521    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:24.031553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.031522    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:24.033953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.033921    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:24.038347 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.038327    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzmh\" (UniqueName: \"kubernetes.io/projected/ee5ba865-e06b-42d3-ba05-fc5093f54f32-kube-api-access-smzmh\") pod \"maas-api-revoke-sub-2779423d-d8p8l\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:24.181412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.181384    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:24.345870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.345764    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:24.474054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.474026    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"]
Nov 03 11:25:24.474174 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:25:24.474025    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5ba865_e06b_42d3_ba05_fc5093f54f32.slice/crio-3b3df12afa899226c3dc384fc6d63144fef5da93a9e96ea63344d6ec44e5a7b7 WatchSource:0}: Error finding container 3b3df12afa899226c3dc384fc6d63144fef5da93a9e96ea63344d6ec44e5a7b7: Status 404 returned error can't find the container with id 3b3df12afa899226c3dc384fc6d63144fef5da93a9e96ea63344d6ec44e5a7b7
Nov 03 11:25:24.893597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:24.884369    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:25.364189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:25.364155    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l" event={"ID":"ee5ba865-e06b-42d3-ba05-fc5093f54f32","Type":"ContainerStarted","Data":"d086071bb6974cf5033d9ded3c29643d55b044ba2f92402a936d794a3dfec2fd"}
Nov 03 11:25:25.364379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:25.364249    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l" event={"ID":"ee5ba865-e06b-42d3-ba05-fc5093f54f32","Type":"ContainerStarted","Data":"3b3df12afa899226c3dc384fc6d63144fef5da93a9e96ea63344d6ec44e5a7b7"}
Nov 03 11:25:25.381660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:25.381616    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l" podStartSLOduration=2.381567132 podStartE2EDuration="2.381567132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:25:25.380226768 +0000 UTC m=+3355.585808820" watchObservedRunningTime="2026-11-03 11:25:25.381567132 +0000 UTC m=+3355.587149183"
Nov 03 11:25:25.579597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:25.572221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:26.256601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:26.251556    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:26.374624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:26.374546    5817 generic.go:356] "Generic (PLEG): container finished" podID="ee5ba865-e06b-42d3-ba05-fc5093f54f32" containerID="d086071bb6974cf5033d9ded3c29643d55b044ba2f92402a936d794a3dfec2fd" exitCode=0
Nov 03 11:25:26.374806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:26.374628    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l" event={"ID":"ee5ba865-e06b-42d3-ba05-fc5093f54f32","Type":"ContainerDied","Data":"d086071bb6974cf5033d9ded3c29643d55b044ba2f92402a936d794a3dfec2fd"}
Nov 03 11:25:26.875061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:26.872620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:27.521180 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.520827    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:27.639033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.638971    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:27.722551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.722521    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5ba865-e06b-42d3-ba05-fc5093f54f32-kube-api-access-smzmh\" (UniqueName: \"kubernetes.io/projected/ee5ba865-e06b-42d3-ba05-fc5093f54f32-kube-api-access-smzmh\") pod \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") "
Nov 03 11:25:27.722784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.722766    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-xks-ca\") pod \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") "
Nov 03 11:25:27.722863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.722798    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-openshift-ca\") pod \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\" (UID: \"ee5ba865-e06b-42d3-ba05-fc5093f54f32\") "
Nov 03 11:25:27.723275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.723237    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-openshift-ca" pod "ee5ba865-e06b-42d3-ba05-fc5093f54f32" (UID: "ee5ba865-e06b-42d3-ba05-fc5093f54f32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:25:27.723779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.723756    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:27.725119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.725090    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-xks-ca" pod "ee5ba865-e06b-42d3-ba05-fc5093f54f32" (UID: "ee5ba865-e06b-42d3-ba05-fc5093f54f32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:25:27.725248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.725186    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5ba865-e06b-42d3-ba05-fc5093f54f32-kube-api-access-smzmh" pod "ee5ba865-e06b-42d3-ba05-fc5093f54f32" (UID: "ee5ba865-e06b-42d3-ba05-fc5093f54f32"). InnerVolumeSpecName "kube-api-access-smzmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:25:27.825941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.825907    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzmh\" (UniqueName: \"kubernetes.io/projected/ee5ba865-e06b-42d3-ba05-fc5093f54f32-kube-api-access-smzmh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:27.826077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:27.825943    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ba865-e06b-42d3-ba05-fc5093f54f32-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:28.164648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:28.164249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:28.404695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:28.404654    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l" event={"ID":"ee5ba865-e06b-42d3-ba05-fc5093f54f32","Type":"ContainerDied","Data":"3b3df12afa899226c3dc384fc6d63144fef5da93a9e96ea63344d6ec44e5a7b7"}
Nov 03 11:25:28.404943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:28.404762    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b3df12afa899226c3dc384fc6d63144fef5da93a9e96ea63344d6ec44e5a7b7"
Nov 03 11:25:28.404943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:28.404871    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"
Nov 03 11:25:28.882103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:28.877108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:29.586603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:29.576448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:30.249099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:30.249059    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:30.687886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:30.687849    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:31.198780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:31.195344    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:31.971320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:31.971283    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:32.667596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:32.664272    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:33.282394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:33.282356    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:33.993366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:33.988871    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:34.767389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:34.767343    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:35.443891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:35.443854    5817 scope.go:122] "RemoveContainer" containerID="247eb96267f98400f83569f71636ce725b2e789dec6e6d218e5e2fd3587741f3"
Nov 03 11:25:35.462046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:35.461911    5817 scope.go:122] "RemoveContainer" containerID="4d253c3e9d7e772e063f1b7dafeccd654d0aa3f0cb0008dc54475d01675b7b3e"
Nov 03 11:25:35.477052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:35.477023    5817 scope.go:122] "RemoveContainer" containerID="e8d48fe03add1353294ce73c48d1466c71e3ec950b953f36dba9d61d9588cc15"
Nov 03 11:25:35.554598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:35.534293    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:36.272515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:36.272417    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:37.064597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:37.064490    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:37.833665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:37.833630    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:38.055631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.055595    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:38.281519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.281485    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"]
Nov 03 11:25:38.282321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.282300    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ee5ba865-e06b-42d3-ba05-fc5093f54f32" containerName="revoke-keys"
Nov 03 11:25:38.282321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.282317    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5ba865-e06b-42d3-ba05-fc5093f54f32" containerName="revoke-keys"
Nov 03 11:25:38.282510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.282429    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5ba865-e06b-42d3-ba05-fc5093f54f32" containerName="revoke-keys"
Nov 03 11:25:38.287951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.287931    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.291504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.291484    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:25:38.291832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.291813    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:25:38.293242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.293215    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"]
Nov 03 11:25:38.375242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.375214    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5ff\" (UniqueName: \"kubernetes.io/projected/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-kube-api-access-gf5ff\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.375421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.375343    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.375471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.375416    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.476730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.476700    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.476884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.476793    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.477011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.476989    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5ff\" (UniqueName: \"kubernetes.io/projected/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-kube-api-access-gf5ff\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.477401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.477378    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.479558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.479536    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.484750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.484725    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5ff\" (UniqueName: \"kubernetes.io/projected/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-kube-api-access-gf5ff\") pod \"maas-api-revoke-sub-53303d76-q9jfs\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.611660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.606227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:38.622597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.613877    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:38.918901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:38.918871    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"]
Nov 03 11:25:38.921100 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:25:38.921071    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e7f6bd_590f_4fe8_9b0a_0125a282f6c0.slice/crio-8eb62a44f28a1275e463de923bb375fbace47ffd9f442e9a8d7125657e016c25 WatchSource:0}: Error finding container 8eb62a44f28a1275e463de923bb375fbace47ffd9f442e9a8d7125657e016c25: Status 404 returned error can't find the container with id 8eb62a44f28a1275e463de923bb375fbace47ffd9f442e9a8d7125657e016c25
Nov 03 11:25:39.351607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:39.349906    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:39.520804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:39.520708    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs" event={"ID":"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0","Type":"ContainerStarted","Data":"3ca756f2cde63e3bde97d3b76185689e9017866664a1d0375373b31b4dcbdca9"}
Nov 03 11:25:39.520804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:39.520763    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs" event={"ID":"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0","Type":"ContainerStarted","Data":"8eb62a44f28a1275e463de923bb375fbace47ffd9f442e9a8d7125657e016c25"}
Nov 03 11:25:39.539989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:39.539931    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs" podStartSLOduration=1.539918388 podStartE2EDuration="1.539918388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:25:39.537863469 +0000 UTC m=+3369.743445518" watchObservedRunningTime="2026-11-03 11:25:39.539918388 +0000 UTC m=+3369.745500442"
Nov 03 11:25:40.161657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:40.156147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:40.530823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:40.530787    5817 generic.go:356] "Generic (PLEG): container finished" podID="01e7f6bd-590f-4fe8-9b0a-0125a282f6c0" containerID="3ca756f2cde63e3bde97d3b76185689e9017866664a1d0375373b31b4dcbdca9" exitCode=0
Nov 03 11:25:40.531022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:40.530870    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs" event={"ID":"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0","Type":"ContainerDied","Data":"3ca756f2cde63e3bde97d3b76185689e9017866664a1d0375373b31b4dcbdca9"}
Nov 03 11:25:40.895719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:40.891527    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:41.584273 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.579170    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:41.814994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.814969    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:41.856293 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.856202    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-xks-ca\") pod \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") "
Nov 03 11:25:41.856607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.856560    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-kube-api-access-gf5ff\" (UniqueName: \"kubernetes.io/projected/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-kube-api-access-gf5ff\") pod \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") "
Nov 03 11:25:41.856713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.856647    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-openshift-ca\") pod \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\" (UID: \"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0\") "
Nov 03 11:25:41.857754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.857721    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-openshift-ca" pod "01e7f6bd-590f-4fe8-9b0a-0125a282f6c0" (UID: "01e7f6bd-590f-4fe8-9b0a-0125a282f6c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:25:41.858936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.858873    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-xks-ca" pod "01e7f6bd-590f-4fe8-9b0a-0125a282f6c0" (UID: "01e7f6bd-590f-4fe8-9b0a-0125a282f6c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:25:41.859905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.859879    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-kube-api-access-gf5ff" pod "01e7f6bd-590f-4fe8-9b0a-0125a282f6c0" (UID: "01e7f6bd-590f-4fe8-9b0a-0125a282f6c0"). InnerVolumeSpecName "kube-api-access-gf5ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:25:41.958983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.958941    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5ff\" (UniqueName: \"kubernetes.io/projected/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-kube-api-access-gf5ff\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:41.958983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.958974    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:41.958983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:41.958990    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:42.269304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:42.267149    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:42.589910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:42.589813    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"
Nov 03 11:25:42.590367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:42.589820    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs" event={"ID":"01e7f6bd-590f-4fe8-9b0a-0125a282f6c0","Type":"ContainerDied","Data":"8eb62a44f28a1275e463de923bb375fbace47ffd9f442e9a8d7125657e016c25"}
Nov 03 11:25:42.590367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:42.589984    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb62a44f28a1275e463de923bb375fbace47ffd9f442e9a8d7125657e016c25"
Nov 03 11:25:42.994612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:42.990526    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:43.766024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:43.765983    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:44.453304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:44.453268    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:45.191033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:45.190996    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:45.986491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:45.986457    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:46.846266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:46.846226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:47.578168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:47.578133    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:48.362815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:48.362780    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:49.033595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.029040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:49.165854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.165817    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"]
Nov 03 11:25:49.166705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.166689    5817 cpu_manager.go:414] "containerMap: removing container" podUID="01e7f6bd-590f-4fe8-9b0a-0125a282f6c0" containerName="revoke-keys"
Nov 03 11:25:49.166757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.166707    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e7f6bd-590f-4fe8-9b0a-0125a282f6c0" containerName="revoke-keys"
Nov 03 11:25:49.166874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.166864    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e7f6bd-590f-4fe8-9b0a-0125a282f6c0" containerName="revoke-keys"
Nov 03 11:25:49.170886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.170865    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.175203 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.175169    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:25:49.175203 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.175178    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:25:49.180427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.180403    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"]
Nov 03 11:25:49.318871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.318781    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dhb\" (UniqueName: \"kubernetes.io/projected/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-kube-api-access-j2dhb\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.332512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.324915    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.332955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.332922    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.334193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.334164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:49.435316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.435284    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.435860 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.435480    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dhb\" (UniqueName: \"kubernetes.io/projected/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-kube-api-access-j2dhb\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.435860 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.435704    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.436248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.436223    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.438344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.438321    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.443489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.443467    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dhb\" (UniqueName: \"kubernetes.io/projected/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-kube-api-access-j2dhb\") pod \"maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.488024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.487998    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:49.772603 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:25:49.772554    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e6d541_f87b_4ce1_807b_4adb2b9d98ce.slice/crio-d75b306a7efc7b28db559f4880c0303da921500a007800bf102ce2518da6c2c0 WatchSource:0}: Error finding container d75b306a7efc7b28db559f4880c0303da921500a007800bf102ce2518da6c2c0: Status 404 returned error can't find the container with id d75b306a7efc7b28db559f4880c0303da921500a007800bf102ce2518da6c2c0
Nov 03 11:25:49.773023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:49.773003    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"]
Nov 03 11:25:50.082626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:50.082507    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:50.670037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:50.670001    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4" event={"ID":"07e6d541-f87b-4ce1-807b-4adb2b9d98ce","Type":"ContainerStarted","Data":"2773534247a9f1321c231c5c6ddefc084a28c3fce03f150e1cf0b55650a1ac97"}
Nov 03 11:25:50.670422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:50.670060    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4" event={"ID":"07e6d541-f87b-4ce1-807b-4adb2b9d98ce","Type":"ContainerStarted","Data":"d75b306a7efc7b28db559f4880c0303da921500a007800bf102ce2518da6c2c0"}
Nov 03 11:25:50.685953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:50.685895    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4" podStartSLOduration=1.6858780599999998 podStartE2EDuration="1.68587806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:25:50.684838731 +0000 UTC m=+3380.890420785" watchObservedRunningTime="2026-11-03 11:25:50.68587806 +0000 UTC m=+3380.891460141"
Nov 03 11:25:50.922664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:50.921108    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:51.707165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:51.707127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:51.711635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:51.710332    5817 generic.go:356] "Generic (PLEG): container finished" podID="07e6d541-f87b-4ce1-807b-4adb2b9d98ce" containerID="2773534247a9f1321c231c5c6ddefc084a28c3fce03f150e1cf0b55650a1ac97" exitCode=0
Nov 03 11:25:51.711635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:51.710546    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4" event={"ID":"07e6d541-f87b-4ce1-807b-4adb2b9d98ce","Type":"ContainerDied","Data":"2773534247a9f1321c231c5c6ddefc084a28c3fce03f150e1cf0b55650a1ac97"}
Nov 03 11:25:52.370598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:52.368160    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:52.428536 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:25:52.428499    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:25:52.912922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:52.912895    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:53.015361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.015269    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-kube-api-access-j2dhb\" (UniqueName: \"kubernetes.io/projected/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-kube-api-access-j2dhb\") pod \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") "
Nov 03 11:25:53.015361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.015348    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-xks-ca\") pod \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") "
Nov 03 11:25:53.015615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.015488    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-openshift-ca\") pod \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\" (UID: \"07e6d541-f87b-4ce1-807b-4adb2b9d98ce\") "
Nov 03 11:25:53.016916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.016890    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-openshift-ca" pod "07e6d541-f87b-4ce1-807b-4adb2b9d98ce" (UID: "07e6d541-f87b-4ce1-807b-4adb2b9d98ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:25:53.018520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.018481    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-kube-api-access-j2dhb" pod "07e6d541-f87b-4ce1-807b-4adb2b9d98ce" (UID: "07e6d541-f87b-4ce1-807b-4adb2b9d98ce"). InnerVolumeSpecName "kube-api-access-j2dhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:25:53.019100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.019074    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-xks-ca" pod "07e6d541-f87b-4ce1-807b-4adb2b9d98ce" (UID: "07e6d541-f87b-4ce1-807b-4adb2b9d98ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:25:53.087619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.084269    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:53.121566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.121517    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.121831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.121812    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dhb\" (UniqueName: \"kubernetes.io/projected/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-kube-api-access-j2dhb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.121929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.121918    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e6d541-f87b-4ce1-807b-4adb2b9d98ce-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.736563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.736527    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"
Nov 03 11:25:53.736842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.736628    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4" event={"ID":"07e6d541-f87b-4ce1-807b-4adb2b9d98ce","Type":"ContainerDied","Data":"d75b306a7efc7b28db559f4880c0303da921500a007800bf102ce2518da6c2c0"}
Nov 03 11:25:53.736842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.736679    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75b306a7efc7b28db559f4880c0303da921500a007800bf102ce2518da6c2c0"
Nov 03 11:25:53.869597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:53.868370    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:54.624592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:54.620452    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:54.770795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:54.770756    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:55.805603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:55.805099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:56.573411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:56.573374    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:57.472709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:57.472603    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:58.306167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:58.306131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:58.988480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:58.988437    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:59.273913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:59.273835    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"]
Nov 03 11:25:59.284890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:59.284859    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"]
Nov 03 11:25:59.394245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:59.394209    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:59.574704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:59.570985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:25:59.726598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:25:59.719171    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:00.418476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.418442    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"]
Nov 03 11:26:00.419392 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.419368    5817 cpu_manager.go:414] "containerMap: removing container" podUID="07e6d541-f87b-4ce1-807b-4adb2b9d98ce" containerName="revoke-keys"
Nov 03 11:26:00.419392 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.419392    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e6d541-f87b-4ce1-807b-4adb2b9d98ce" containerName="revoke-keys"
Nov 03 11:26:00.419554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.419517    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e6d541-f87b-4ce1-807b-4adb2b9d98ce" containerName="revoke-keys"
Nov 03 11:26:00.430591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.426535    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.430591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.430310    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:26:00.431738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.431713    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:26:00.432260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.432241    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"]
Nov 03 11:26:00.490619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.490585    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.490769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.490660    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnpf\" (UniqueName: \"kubernetes.io/projected/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-kube-api-access-nlnpf\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.490877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.490863    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.592369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.592329    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.592600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.592427    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnpf\" (UniqueName: \"kubernetes.io/projected/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-kube-api-access-nlnpf\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.592687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.592650    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.593409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.593380    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.595326 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.595298    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.603565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.603540    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnpf\" (UniqueName: \"kubernetes.io/projected/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-kube-api-access-nlnpf\") pod \"maas-api-revoke-sub-7f2a3b3e-82mk2\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:00.643435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.633220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:00.752196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:00.752168    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:01.042225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:01.042190    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"]
Nov 03 11:26:01.043841 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:26:01.043813    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7e1d97_0679_4530_b888_fa0f86b5d3a4.slice/crio-1320beec09a5b8d72a4242fc72d3f026899dd7bae9fb8bcf5087e96dab4a0fa6 WatchSource:0}: Error finding container 1320beec09a5b8d72a4242fc72d3f026899dd7bae9fb8bcf5087e96dab4a0fa6: Status 404 returned error can't find the container with id 1320beec09a5b8d72a4242fc72d3f026899dd7bae9fb8bcf5087e96dab4a0fa6
Nov 03 11:26:01.253596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:01.253507    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:01.896596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:01.895199    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:01.959926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:01.959879    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2" event={"ID":"4b7e1d97-0679-4530-b888-fa0f86b5d3a4","Type":"ContainerStarted","Data":"4c5bc212f8e8da7f1fdb9e5b0bd02fe354432a974b49e5121db2dabb7c5bea09"}
Nov 03 11:26:01.960233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:01.960214    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2" event={"ID":"4b7e1d97-0679-4530-b888-fa0f86b5d3a4","Type":"ContainerStarted","Data":"1320beec09a5b8d72a4242fc72d3f026899dd7bae9fb8bcf5087e96dab4a0fa6"}
Nov 03 11:26:01.969404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:01.969361    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2" podStartSLOduration=1.969348599 podStartE2EDuration="1.969348599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:26:01.967436863 +0000 UTC m=+3392.173018915" watchObservedRunningTime="2026-11-03 11:26:01.969348599 +0000 UTC m=+3392.174930650"
Nov 03 11:26:02.690254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:02.681261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:02.954552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:02.954465    5817 generic.go:356] "Generic (PLEG): container finished" podID="4b7e1d97-0679-4530-b888-fa0f86b5d3a4" containerID="4c5bc212f8e8da7f1fdb9e5b0bd02fe354432a974b49e5121db2dabb7c5bea09" exitCode=0
Nov 03 11:26:02.954552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:02.954503    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2" event={"ID":"4b7e1d97-0679-4530-b888-fa0f86b5d3a4","Type":"ContainerDied","Data":"4c5bc212f8e8da7f1fdb9e5b0bd02fe354432a974b49e5121db2dabb7c5bea09"}
Nov 03 11:26:03.563537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:03.563503    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:04.155757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.155736    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:04.185707 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.185657    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-xks-ca\") pod \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") "
Nov 03 11:26:04.186052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.186029    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-openshift-ca\") pod \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") "
Nov 03 11:26:04.186158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.186073    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-kube-api-access-nlnpf\" (UniqueName: \"kubernetes.io/projected/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-kube-api-access-nlnpf\") pod \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\" (UID: \"4b7e1d97-0679-4530-b888-fa0f86b5d3a4\") "
Nov 03 11:26:04.187005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.186973    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-openshift-ca" pod "4b7e1d97-0679-4530-b888-fa0f86b5d3a4" (UID: "4b7e1d97-0679-4530-b888-fa0f86b5d3a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:04.188719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.188682    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-xks-ca" pod "4b7e1d97-0679-4530-b888-fa0f86b5d3a4" (UID: "4b7e1d97-0679-4530-b888-fa0f86b5d3a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:26:04.192396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.192362    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-kube-api-access-nlnpf" pod "4b7e1d97-0679-4530-b888-fa0f86b5d3a4" (UID: "4b7e1d97-0679-4530-b888-fa0f86b5d3a4"). InnerVolumeSpecName "kube-api-access-nlnpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:04.267658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.265469    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:04.307331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.301671    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq" podUID="43d55c2c-2972-4fcf-96ca-0629f294a1b1" containerName="payload-processing" containerID="cri-o://d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6" gracePeriod=25
Nov 03 11:26:04.321719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.315997    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s" podUID="838aae41-6959-4c5d-b1a8-348ca9375a72" containerName="payload-pre-processing" containerID="cri-o://04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5" gracePeriod=25
Nov 03 11:26:04.329667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.326872    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:04.329667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.326903    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:04.329667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.326927    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlnpf\" (UniqueName: \"kubernetes.io/projected/4b7e1d97-0679-4530-b888-fa0f86b5d3a4-kube-api-access-nlnpf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:04.905597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.895868    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:04.972904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.972877    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"
Nov 03 11:26:04.973056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.972899    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2" event={"ID":"4b7e1d97-0679-4530-b888-fa0f86b5d3a4","Type":"ContainerDied","Data":"1320beec09a5b8d72a4242fc72d3f026899dd7bae9fb8bcf5087e96dab4a0fa6"}
Nov 03 11:26:04.973056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:04.972945    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1320beec09a5b8d72a4242fc72d3f026899dd7bae9fb8bcf5087e96dab4a0fa6"
Nov 03 11:26:05.486602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:05.485432    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:06.115597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:06.114285    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:06.845478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:06.845446    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:07.449603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:07.448337    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:08.055594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:08.050179    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:08.595408 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:08.595368    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:08.724036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:08.724003    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:09.025205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:09.025176    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"]
Nov 03 11:26:09.033099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:09.033072    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d851a1a-5c4bf"]
Nov 03 11:26:09.348212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:09.348122    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:09.904525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:09.904494    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e74348c-5720-4bc9-ad8b-5cf35a7cc2da" path="/var/lib/kubelet/pods/1e74348c-5720-4bc9-ad8b-5cf35a7cc2da/volumes"
Nov 03 11:26:10.046368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:10.046330    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:10.625611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:10.625541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:10.794641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:10.777138    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:11.028238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:11.028198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:11.049289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:11.049236    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"]
Nov 03 11:26:11.058478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:11.058445    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b13404-fp4j4"]
Nov 03 11:26:11.243495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:11.243458    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:11.446785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:11.446754    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:11.903526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:11.903450    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e6d541-f87b-4ce1-807b-4adb2b9d98ce" path="/var/lib/kubelet/pods/07e6d541-f87b-4ce1-807b-4adb2b9d98ce/volumes"
Nov 03 11:26:12.108356 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:12.108325    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:12.674003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:12.673949    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:12.768671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:12.768635    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s" podUID="838aae41-6959-4c5d-b1a8-348ca9375a72" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:26:13.021781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:13.021753    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"]
Nov 03 11:26:13.037506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:13.037476    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"]
Nov 03 11:26:13.139415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:13.139332    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:13.344562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:13.344474    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:13.672683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:13.666109    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:13.779872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:13.779832    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq" podUID="43d55c2c-2972-4fcf-96ca-0629f294a1b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:26:14.349598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:14.345068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:14.984616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:14.984559    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:15.444129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:15.444096    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:15.640596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:15.640139    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:15.877662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:15.875182    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:16.244596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:16.238239    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:16.443470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:16.443433    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:17.230335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:17.230296    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:17.438859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:17.438828    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:17.859673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:17.859633    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:18.022238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.022195    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh" podUID="c83f7c03-e465-4d5c-80e8-ae88da2976a9" containerName="payload-processing" containerID="cri-o://7a67c8865c4006d2ec2c10fca51bdba688b9672d5d249ec736b7d522a56fde51" gracePeriod=25
Nov 03 11:26:18.047558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.047531    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:18.152540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.152488    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx" podUID="91d1b8bb-edae-4b26-9efc-0418a566a48d" containerName="payload-pre-processing" containerID="cri-o://f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517" gracePeriod=25
Nov 03 11:26:18.189600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.189555    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:18.249294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.249143    5817 generic.go:356] "Generic (PLEG): container finished" podID="c83f7c03-e465-4d5c-80e8-ae88da2976a9" containerID="7a67c8865c4006d2ec2c10fca51bdba688b9672d5d249ec736b7d522a56fde51" exitCode=0
Nov 03 11:26:18.249294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.249291    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh" event={"ID":"c83f7c03-e465-4d5c-80e8-ae88da2976a9","Type":"ContainerDied","Data":"7a67c8865c4006d2ec2c10fca51bdba688b9672d5d249ec736b7d522a56fde51"}
Nov 03 11:26:18.298172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.298143    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"]
Nov 03 11:26:18.298526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.298426    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy" containerID="cri-o://5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67" gracePeriod=30
Nov 03 11:26:18.369526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.369297    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:18.651726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.546982    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:26:18.688602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.681856    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83f7c03-e465-4d5c-80e8-ae88da2976a9-kube-api-access-hq8mh\" (UniqueName: \"kubernetes.io/projected/c83f7c03-e465-4d5c-80e8-ae88da2976a9-kube-api-access-hq8mh\") pod \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\" (UID: \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\") "
Nov 03 11:26:18.688602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.682477    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83f7c03-e465-4d5c-80e8-ae88da2976a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f7c03-e465-4d5c-80e8-ae88da2976a9-plugins-config-volume\") pod \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\" (UID: \"c83f7c03-e465-4d5c-80e8-ae88da2976a9\") "
Nov 03 11:26:18.695505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.691077    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83f7c03-e465-4d5c-80e8-ae88da2976a9-plugins-config-volume" pod "c83f7c03-e465-4d5c-80e8-ae88da2976a9" (UID: "c83f7c03-e465-4d5c-80e8-ae88da2976a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:18.705623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.704945    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83f7c03-e465-4d5c-80e8-ae88da2976a9-kube-api-access-hq8mh" pod "c83f7c03-e465-4d5c-80e8-ae88da2976a9" (UID: "c83f7c03-e465-4d5c-80e8-ae88da2976a9"). InnerVolumeSpecName "kube-api-access-hq8mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:18.729159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.729128    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:26:18.782264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.782234    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"]
Nov 03 11:26:18.783142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783119    5817 cpu_manager.go:414] "containerMap: removing container" podUID="91d1b8bb-edae-4b26-9efc-0418a566a48d" containerName="payload-pre-processing"
Nov 03 11:26:18.783142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783141    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d1b8bb-edae-4b26-9efc-0418a566a48d" containerName="payload-pre-processing"
Nov 03 11:26:18.783337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783174    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4b7e1d97-0679-4530-b888-fa0f86b5d3a4" containerName="revoke-keys"
Nov 03 11:26:18.783337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783182    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7e1d97-0679-4530-b888-fa0f86b5d3a4" containerName="revoke-keys"
Nov 03 11:26:18.783337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783201    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c83f7c03-e465-4d5c-80e8-ae88da2976a9" containerName="payload-processing"
Nov 03 11:26:18.783337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783209    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83f7c03-e465-4d5c-80e8-ae88da2976a9" containerName="payload-processing"
Nov 03 11:26:18.783554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783393    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83f7c03-e465-4d5c-80e8-ae88da2976a9" containerName="payload-processing"
Nov 03 11:26:18.783554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783424    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d1b8bb-edae-4b26-9efc-0418a566a48d" containerName="payload-pre-processing"
Nov 03 11:26:18.783554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.783437    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7e1d97-0679-4530-b888-fa0f86b5d3a4" containerName="revoke-keys"
Nov 03 11:26:18.787292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.787250    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:18.791336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.791282    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:26:18.792418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.791863    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:26:18.792542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.792477    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8mh\" (UniqueName: \"kubernetes.io/projected/c83f7c03-e465-4d5c-80e8-ae88da2976a9-kube-api-access-hq8mh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:18.792542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.792505    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f7c03-e465-4d5c-80e8-ae88da2976a9-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:18.794196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.794175    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"]
Nov 03 11:26:18.882596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.882485    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:18.904645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.904250    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d1b8bb-edae-4b26-9efc-0418a566a48d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d1b8bb-edae-4b26-9efc-0418a566a48d-plugins-config-volume\") pod \"91d1b8bb-edae-4b26-9efc-0418a566a48d\" (UID: \"91d1b8bb-edae-4b26-9efc-0418a566a48d\") "
Nov 03 11:26:18.904645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.904305    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d1b8bb-edae-4b26-9efc-0418a566a48d-kube-api-access-ljtsv\" (UniqueName: \"kubernetes.io/projected/91d1b8bb-edae-4b26-9efc-0418a566a48d-kube-api-access-ljtsv\") pod \"91d1b8bb-edae-4b26-9efc-0418a566a48d\" (UID: \"91d1b8bb-edae-4b26-9efc-0418a566a48d\") "
Nov 03 11:26:18.904933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.904826    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:18.904933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.904891    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgn8\" (UniqueName: \"kubernetes.io/projected/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-kube-api-access-zbgn8\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:18.913118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.905111    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:18.913118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.905687    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d1b8bb-edae-4b26-9efc-0418a566a48d-plugins-config-volume" pod "91d1b8bb-edae-4b26-9efc-0418a566a48d" (UID: "91d1b8bb-edae-4b26-9efc-0418a566a48d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:18.913118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:18.909302    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d1b8bb-edae-4b26-9efc-0418a566a48d-kube-api-access-ljtsv" pod "91d1b8bb-edae-4b26-9efc-0418a566a48d" (UID: "91d1b8bb-edae-4b26-9efc-0418a566a48d"). InnerVolumeSpecName "kube-api-access-ljtsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:19.006649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.006610    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:19.006824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.006668    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgn8\" (UniqueName: \"kubernetes.io/projected/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-kube-api-access-zbgn8\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:19.006824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.006797    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:19.006999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.006987    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d1b8bb-edae-4b26-9efc-0418a566a48d-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:19.007057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.007002    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljtsv\" (UniqueName: \"kubernetes.io/projected/91d1b8bb-edae-4b26-9efc-0418a566a48d-kube-api-access-ljtsv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:19.007415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.007391    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:19.009391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.009365    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:19.014542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.014523    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgn8\" (UniqueName: \"kubernetes.io/projected/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-kube-api-access-zbgn8\") pod \"maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:19.017142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.017115    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:19.106727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.106703    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:19.259439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.259405    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"
Nov 03 11:26:19.259900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.259438    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh" event={"ID":"c83f7c03-e465-4d5c-80e8-ae88da2976a9","Type":"ContainerDied","Data":"31a73c320733ce11555a9f98b67c5f0b5201718a119c26077257916b8115de0f"}
Nov 03 11:26:19.259900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.259495    5817 scope.go:122] "RemoveContainer" containerID="7a67c8865c4006d2ec2c10fca51bdba688b9672d5d249ec736b7d522a56fde51"
Nov 03 11:26:19.261165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.261100    5817 generic.go:356] "Generic (PLEG): container finished" podID="91d1b8bb-edae-4b26-9efc-0418a566a48d" containerID="f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517" exitCode=0
Nov 03 11:26:19.261272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.261158    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx" event={"ID":"91d1b8bb-edae-4b26-9efc-0418a566a48d","Type":"ContainerDied","Data":"f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517"}
Nov 03 11:26:19.261272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.261207    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx" event={"ID":"91d1b8bb-edae-4b26-9efc-0418a566a48d","Type":"ContainerDied","Data":"a9401671cd5ecbd94cd3b05f8c1a734cf6d942a1348afc934aa09d80362c0dbd"}
Nov 03 11:26:19.261272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.261137    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"
Nov 03 11:26:19.272322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.272250    5817 scope.go:122] "RemoveContainer" containerID="f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517"
Nov 03 11:26:19.283178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.283161    5817 scope.go:122] "RemoveContainer" containerID="f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517"
Nov 03 11:26:19.283456 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:19.283439    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517\": container with ID starting with f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517 not found: ID does not exist" containerID="f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517"
Nov 03 11:26:19.283509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.283463    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517"} err="failed to get container status \"f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517\": rpc error: code = NotFound desc = could not find container \"f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517\": container with ID starting with f62e7819cee8961c5b82450000d08a00520286545bfa9a61a4a17314acbd2517 not found: ID does not exist"
Nov 03 11:26:19.291726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.291701    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"]
Nov 03 11:26:19.296386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.296363    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-049d3b4c-6f95dfffdf-h6dqh"]
Nov 03 11:26:19.306897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.306875    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"]
Nov 03 11:26:19.311220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.311202    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-049d3b4c-54d6b698d7-mrcpx"]
Nov 03 11:26:19.392787 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:26:19.392762    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dab1729_3db2_46bf_80c8_2ec3b4b57e5c.slice/crio-51ab88f854a51835ae766041168dc09501853404e76368c215073efd40662354 WatchSource:0}: Error finding container 51ab88f854a51835ae766041168dc09501853404e76368c215073efd40662354: Status 404 returned error can't find the container with id 51ab88f854a51835ae766041168dc09501853404e76368c215073efd40662354
Nov 03 11:26:19.393846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.393819    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"]
Nov 03 11:26:19.755797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.755751    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:19.903471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.903444    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d1b8bb-edae-4b26-9efc-0418a566a48d" path="/var/lib/kubelet/pods/91d1b8bb-edae-4b26-9efc-0418a566a48d/volumes"
Nov 03 11:26:19.903949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:19.903930    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83f7c03-e465-4d5c-80e8-ae88da2976a9" path="/var/lib/kubelet/pods/c83f7c03-e465-4d5c-80e8-ae88da2976a9/volumes"
Nov 03 11:26:20.273344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:20.273308    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l" event={"ID":"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c","Type":"ContainerStarted","Data":"f186c04e28d2e4868fec42ac50f14003ded04848e6397d31391a6e560b43de28"}
Nov 03 11:26:20.273965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:20.273361    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l" event={"ID":"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c","Type":"ContainerStarted","Data":"51ab88f854a51835ae766041168dc09501853404e76368c215073efd40662354"}
Nov 03 11:26:20.291249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:20.291172    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l" podStartSLOduration=2.29115271 podStartE2EDuration="2.29115271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:26:20.288444542 +0000 UTC m=+3410.494026595" watchObservedRunningTime="2026-11-03 11:26:20.29115271 +0000 UTC m=+3410.496734756"
Nov 03 11:26:20.461605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:20.461014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:21.022512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:21.022481    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"]
Nov 03 11:26:21.027455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:21.027430    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eedce5f-8g4cp"]
Nov 03 11:26:21.193596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:21.191226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:21.286969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:21.286882    5817 generic.go:356] "Generic (PLEG): container finished" podID="7dab1729-3db2-46bf-80c8-2ec3b4b57e5c" containerID="f186c04e28d2e4868fec42ac50f14003ded04848e6397d31391a6e560b43de28" exitCode=0
Nov 03 11:26:21.287374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:21.286971    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l" event={"ID":"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c","Type":"ContainerDied","Data":"f186c04e28d2e4868fec42ac50f14003ded04848e6397d31391a6e560b43de28"}
Nov 03 11:26:21.860914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:21.860874    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:21.904610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:21.904556    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c84c58-b971-4029-bf01-734c5bc1cab6" path="/var/lib/kubelet/pods/96c84c58-b971-4029-bf01-734c5bc1cab6/volumes"
Nov 03 11:26:22.364827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.364697    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:22.486330 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:22.486294    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:26:22.514622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.513857    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:22.516717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.516685    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:22.617784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.617686    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-openshift-ca\") pod \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") "
Nov 03 11:26:22.617965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.617891    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-xks-ca\") pod \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") "
Nov 03 11:26:22.618060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.618045    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-kube-api-access-zbgn8\" (UniqueName: \"kubernetes.io/projected/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-kube-api-access-zbgn8\") pod \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\" (UID: \"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c\") "
Nov 03 11:26:22.618128 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.618093    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-openshift-ca" pod "7dab1729-3db2-46bf-80c8-2ec3b4b57e5c" (UID: "7dab1729-3db2-46bf-80c8-2ec3b4b57e5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:22.618828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.618803    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:22.622216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.622187    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-kube-api-access-zbgn8" pod "7dab1729-3db2-46bf-80c8-2ec3b4b57e5c" (UID: "7dab1729-3db2-46bf-80c8-2ec3b4b57e5c"). InnerVolumeSpecName "kube-api-access-zbgn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:22.622356 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.622282    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-xks-ca" pod "7dab1729-3db2-46bf-80c8-2ec3b4b57e5c" (UID: "7dab1729-3db2-46bf-80c8-2ec3b4b57e5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:26:22.665221 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.665190    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:22.727598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.727373    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgn8\" (UniqueName: \"kubernetes.io/projected/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-kube-api-access-zbgn8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:22.727598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.727419    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:22.811665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.807674    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s" podUID="838aae41-6959-4c5d-b1a8-348ca9375a72" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:26:22.823595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:22.820205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:23.306490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.306463    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"
Nov 03 11:26:23.306737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.306492    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l" event={"ID":"7dab1729-3db2-46bf-80c8-2ec3b4b57e5c","Type":"ContainerDied","Data":"51ab88f854a51835ae766041168dc09501853404e76368c215073efd40662354"}
Nov 03 11:26:23.306737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.306548    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51ab88f854a51835ae766041168dc09501853404e76368c215073efd40662354"
Nov 03 11:26:23.485169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.485124    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:23.698238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.698211    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:26:23.781162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.781120    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq" podUID="43d55c2c-2972-4fcf-96ca-0629f294a1b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:26:23.851521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851436    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-token\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-token\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.851521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851506    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-kube-api-access-qnpkz\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-kube-api-access-qnpkz\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.851802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851540    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-podinfo\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.851802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851642    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-credential-socket\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.851802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851668    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-socket\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.851802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851687    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-certs\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.851802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851741    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-envoy\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.852119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851926    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-data\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.852119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.851993    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-credential-socket" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.852119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852005    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-socket" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.852119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852036    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99eda1c3-5954-45d0-b159-d1f95545bd75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99eda1c3-5954-45d0-b159-d1f95545bd75-istiod-ca-cert\") pod \"99eda1c3-5954-45d0-b159-d1f95545bd75\" (UID: \"99eda1c3-5954-45d0-b159-d1f95545bd75\") "
Nov 03 11:26:23.852352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852149    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-certs" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.852352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852212    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-data" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.852702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852683    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.852773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852704    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.852773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852714    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.852773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852722    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.852956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.852792    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99eda1c3-5954-45d0-b159-d1f95545bd75-istiod-ca-cert" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:23.854136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.854108    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-podinfo" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:26:23.854136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.854129    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-envoy" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.854442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.854424    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-token" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:23.854518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.854456    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-kube-api-access-qnpkz" pod "99eda1c3-5954-45d0-b159-d1f95545bd75" (UID: "99eda1c3-5954-45d0-b159-d1f95545bd75"). InnerVolumeSpecName "kube-api-access-qnpkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:23.953818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.953775    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99eda1c3-5954-45d0-b159-d1f95545bd75-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.953818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.953813    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.953818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.953825    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnpkz\" (UniqueName: \"kubernetes.io/projected/99eda1c3-5954-45d0-b159-d1f95545bd75-kube-api-access-qnpkz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.953818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.953834    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.954158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:23.953844    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99eda1c3-5954-45d0-b159-d1f95545bd75-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:24.163145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.162023    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:24.322949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.322909    5817 generic.go:356] "Generic (PLEG): container finished" podID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerID="5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67" exitCode=0
Nov 03 11:26:24.323150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.322960    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" event={"ID":"99eda1c3-5954-45d0-b159-d1f95545bd75","Type":"ContainerDied","Data":"5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67"}
Nov 03 11:26:24.323150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.322990    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"
Nov 03 11:26:24.323150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.323029    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85" event={"ID":"99eda1c3-5954-45d0-b159-d1f95545bd75","Type":"ContainerDied","Data":"fbe4764091b43c44cdb6b94f2efa59f3f306297b1da45f6ba895ae263ce3e47d"}
Nov 03 11:26:24.323150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.323063    5817 scope.go:122] "RemoveContainer" containerID="5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67"
Nov 03 11:26:24.336718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.336705    5817 scope.go:122] "RemoveContainer" containerID="98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05"
Nov 03 11:26:24.346244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.346216    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"]
Nov 03 11:26:24.351020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.350996    5817 scope.go:122] "RemoveContainer" containerID="5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67"
Nov 03 11:26:24.351370 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:24.351337    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67\": container with ID starting with 5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67 not found: ID does not exist" containerID="5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67"
Nov 03 11:26:24.351455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.351383    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67"} err="failed to get container status \"5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67\": rpc error: code = NotFound desc = could not find container \"5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67\": container with ID starting with 5fd0c452ec5a1a23810a7dae5a36d4edd885f3a75896bad0d6a041be2b6a4f67 not found: ID does not exist"
Nov 03 11:26:24.351455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.351407    5817 scope.go:122] "RemoveContainer" containerID="98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05"
Nov 03 11:26:24.351621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.351472    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-7rp85"]
Nov 03 11:26:24.351710 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:24.351691    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05\": container with ID starting with 98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05 not found: ID does not exist" containerID="98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05"
Nov 03 11:26:24.351769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.351716    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05"} err="failed to get container status \"98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05\": rpc error: code = NotFound desc = could not find container \"98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05\": container with ID starting with 98fc8fa89be39f8702ef36759a6d55aea101519efa6bf6b5cacf728d7928ca05 not found: ID does not exist"
Nov 03 11:26:24.633010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.632978    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:26:24.661548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.661519    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:26:24.800246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.795562    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838aae41-6959-4c5d-b1a8-348ca9375a72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838aae41-6959-4c5d-b1a8-348ca9375a72-plugins-config-volume\") pod \"838aae41-6959-4c5d-b1a8-348ca9375a72\" (UID: \"838aae41-6959-4c5d-b1a8-348ca9375a72\") "
Nov 03 11:26:24.800246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.795698    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d55c2c-2972-4fcf-96ca-0629f294a1b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d55c2c-2972-4fcf-96ca-0629f294a1b1-plugins-config-volume\") pod \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\" (UID: \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\") "
Nov 03 11:26:24.800246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.795837    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d55c2c-2972-4fcf-96ca-0629f294a1b1-kube-api-access-5kdtd\" (UniqueName: \"kubernetes.io/projected/43d55c2c-2972-4fcf-96ca-0629f294a1b1-kube-api-access-5kdtd\") pod \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\" (UID: \"43d55c2c-2972-4fcf-96ca-0629f294a1b1\") "
Nov 03 11:26:24.800246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.796375    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838aae41-6959-4c5d-b1a8-348ca9375a72-kube-api-access-vk74t\" (UniqueName: \"kubernetes.io/projected/838aae41-6959-4c5d-b1a8-348ca9375a72-kube-api-access-vk74t\") pod \"838aae41-6959-4c5d-b1a8-348ca9375a72\" (UID: \"838aae41-6959-4c5d-b1a8-348ca9375a72\") "
Nov 03 11:26:24.800246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.796776    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d55c2c-2972-4fcf-96ca-0629f294a1b1-plugins-config-volume" pod "43d55c2c-2972-4fcf-96ca-0629f294a1b1" (UID: "43d55c2c-2972-4fcf-96ca-0629f294a1b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:24.800246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.797164    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d55c2c-2972-4fcf-96ca-0629f294a1b1-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:24.800246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.797460    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838aae41-6959-4c5d-b1a8-348ca9375a72-plugins-config-volume" pod "838aae41-6959-4c5d-b1a8-348ca9375a72" (UID: "838aae41-6959-4c5d-b1a8-348ca9375a72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:24.812071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.812027    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838aae41-6959-4c5d-b1a8-348ca9375a72-kube-api-access-vk74t" pod "838aae41-6959-4c5d-b1a8-348ca9375a72" (UID: "838aae41-6959-4c5d-b1a8-348ca9375a72"). InnerVolumeSpecName "kube-api-access-vk74t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:24.812688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.812656    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d55c2c-2972-4fcf-96ca-0629f294a1b1-kube-api-access-5kdtd" pod "43d55c2c-2972-4fcf-96ca-0629f294a1b1" (UID: "43d55c2c-2972-4fcf-96ca-0629f294a1b1"). InnerVolumeSpecName "kube-api-access-5kdtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:24.828910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.828879    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:24.898802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.898767    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838aae41-6959-4c5d-b1a8-348ca9375a72-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:24.898802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.898794    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kdtd\" (UniqueName: \"kubernetes.io/projected/43d55c2c-2972-4fcf-96ca-0629f294a1b1-kube-api-access-5kdtd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:24.898802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:24.898804    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk74t\" (UniqueName: \"kubernetes.io/projected/838aae41-6959-4c5d-b1a8-348ca9375a72-kube-api-access-vk74t\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:25.336173 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.336142    5817 generic.go:356] "Generic (PLEG): container finished" podID="838aae41-6959-4c5d-b1a8-348ca9375a72" containerID="04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5" exitCode=0
Nov 03 11:26:25.336393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.336225    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"
Nov 03 11:26:25.336393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.336226    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s" event={"ID":"838aae41-6959-4c5d-b1a8-348ca9375a72","Type":"ContainerDied","Data":"04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5"}
Nov 03 11:26:25.336393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.336317    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s" event={"ID":"838aae41-6959-4c5d-b1a8-348ca9375a72","Type":"ContainerDied","Data":"858220e75dc7a4acb47f643316ea74861cd9d2b2cfcec32ecffdb804abfea3e7"}
Nov 03 11:26:25.336393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.336351    5817 scope.go:122] "RemoveContainer" containerID="04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5"
Nov 03 11:26:25.337747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.337722    5817 generic.go:356] "Generic (PLEG): container finished" podID="43d55c2c-2972-4fcf-96ca-0629f294a1b1" containerID="d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6" exitCode=0
Nov 03 11:26:25.337882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.337751    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq" event={"ID":"43d55c2c-2972-4fcf-96ca-0629f294a1b1","Type":"ContainerDied","Data":"d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6"}
Nov 03 11:26:25.337882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.337788    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"
Nov 03 11:26:25.337882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.337797    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq" event={"ID":"43d55c2c-2972-4fcf-96ca-0629f294a1b1","Type":"ContainerDied","Data":"9ae201c2935808867cac7111b0bede26fadf70a92eb4a84f7462c198b4df9044"}
Nov 03 11:26:25.349911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.349892    5817 scope.go:122] "RemoveContainer" containerID="04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5"
Nov 03 11:26:25.350189 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:25.350171    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5\": container with ID starting with 04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5 not found: ID does not exist" containerID="04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5"
Nov 03 11:26:25.350262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.350197    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5"} err="failed to get container status \"04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5\": rpc error: code = NotFound desc = could not find container \"04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5\": container with ID starting with 04e89b4c72eeaec7eef0798d543e1f270031adaf214e14bf0bafb516c623b2e5 not found: ID does not exist"
Nov 03 11:26:25.350262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.350211    5817 scope.go:122] "RemoveContainer" containerID="d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6"
Nov 03 11:26:25.361994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.361974    5817 scope.go:122] "RemoveContainer" containerID="d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6"
Nov 03 11:26:25.362336 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:25.362296    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6\": container with ID starting with d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6 not found: ID does not exist" containerID="d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6"
Nov 03 11:26:25.362406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.362345    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6"} err="failed to get container status \"d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6\": rpc error: code = NotFound desc = could not find container \"d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6\": container with ID starting with d2f68a9ed961f4410485574e91641b9ed876ee6996f3b644616fa8d83d1cf5e6 not found: ID does not exist"
Nov 03 11:26:25.367516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.367492    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"]
Nov 03 11:26:25.372010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.371988    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b13404-f996d75f-xd57s"]
Nov 03 11:26:25.384208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.384190    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"]
Nov 03 11:26:25.388964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.388943    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b13404-766b656b97-mngzq"]
Nov 03 11:26:25.472256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.469641    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:25.906977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.906944    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d55c2c-2972-4fcf-96ca-0629f294a1b1" path="/var/lib/kubelet/pods/43d55c2c-2972-4fcf-96ca-0629f294a1b1/volumes"
Nov 03 11:26:25.907426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.907412    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="838aae41-6959-4c5d-b1a8-348ca9375a72" path="/var/lib/kubelet/pods/838aae41-6959-4c5d-b1a8-348ca9375a72/volumes"
Nov 03 11:26:25.907880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:25.907866    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" path="/var/lib/kubelet/pods/99eda1c3-5954-45d0-b159-d1f95545bd75/volumes"
Nov 03 11:26:26.195031 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:26.190383    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:26.834602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:26.834469    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:27.502005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:27.499678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:28.289461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:28.289422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:28.909189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:28.908766    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"]
Nov 03 11:26:29.023485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:29.023449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:29.167209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:29.167121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:29.762594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:29.760276    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:30.473600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:30.467678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:31.180916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:31.180880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:31.780159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:31.780118    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:32.480301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:32.480264    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:33.156595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:33.155549    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:33.766427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:33.766388    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:33.909638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:33.909552    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9" podUID="be9299d8-3824-4025-a353-119470c37edd" containerName="payload-pre-processing" containerID="cri-o://e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65" gracePeriod=25
Nov 03 11:26:34.422043 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:34.422007    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:34.992529 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:34.992496    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:26:35.027771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:35.027740    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"]
Nov 03 11:26:35.036636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:35.036606    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0cff11-l4h42"]
Nov 03 11:26:35.091597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:35.087372    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:35.839466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:35.839375    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:35.972725 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:35.960915    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013cd32c-45d0-4685-a9cd-bbeb9ab78bea" path="/var/lib/kubelet/pods/013cd32c-45d0-4685-a9cd-bbeb9ab78bea/volumes"
Nov 03 11:26:36.843137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:36.843105    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:37.805838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:37.805798    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:37.950981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:37.950935    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:37.996652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:37.989518    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:37.998717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:37.998676    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:38.031527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.031481    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:38.032167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.031648    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw" podUID="0a480b85-cd03-4052-a431-4d092fa8600b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 11:26:38.127799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.127658    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:38.237846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.237800    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-c7n4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:26:38.238059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.237881    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-c7n4c" podUID="edf3ef4f-1840-4e68-9049-ec8a195d5e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:38.309608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.298366    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 03 11:26:38.434595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.434534    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:38.466702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.466653    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:38.567704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.567649    5817 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:38.591694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.591652    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:38.742842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.742748    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:38.945741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.937668    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7hrjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:26:38.945741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:38.937729    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:26:39.493671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:39.488094    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:40.165596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:40.162735    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:40.767886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:40.767846    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:41.374412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:41.374351    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:42.176600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.170341    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:42.398717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.398673    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9" podUID="be9299d8-3824-4025-a353-119470c37edd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:26:42.567747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.567712    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"]
Nov 03 11:26:42.567975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.567942    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5" podUID="2620222d-9f11-40c4-b27d-e15d59be1755" containerName="maas-api" containerID="cri-o://682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a" gracePeriod=30
Nov 03 11:26:42.652059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.652028    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5" podUID="2620222d-9f11-40c4-b27d-e15d59be1755" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.82:8443/health\": dial tcp 10.133.1.82:8443: connect: connection refused"
Nov 03 11:26:42.656506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.656477    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:42.808742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.808677    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"]
Nov 03 11:26:42.816028 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.815998    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9143e1-2zr8l"]
Nov 03 11:26:42.831175 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.831109    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:42.933240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:42.932886    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:26:43.013419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.013314    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:43.072420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.072204    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-maas-api-tls\") pod \"2620222d-9f11-40c4-b27d-e15d59be1755\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") "
Nov 03 11:26:43.072944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.072879    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2620222d-9f11-40c4-b27d-e15d59be1755-kube-api-access-5g5bd\" (UniqueName: \"kubernetes.io/projected/2620222d-9f11-40c4-b27d-e15d59be1755-kube-api-access-5g5bd\") pod \"2620222d-9f11-40c4-b27d-e15d59be1755\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") "
Nov 03 11:26:43.073323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.073303    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-metrics-certs\") pod \"2620222d-9f11-40c4-b27d-e15d59be1755\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") "
Nov 03 11:26:43.073891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.073845    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2620222d-9f11-40c4-b27d-e15d59be1755-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2620222d-9f11-40c4-b27d-e15d59be1755-openshift-service-ca\") pod \"2620222d-9f11-40c4-b27d-e15d59be1755\" (UID: \"2620222d-9f11-40c4-b27d-e15d59be1755\") "
Nov 03 11:26:43.082137 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.082066    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2620222d-9f11-40c4-b27d-e15d59be1755-openshift-service-ca" pod "2620222d-9f11-40c4-b27d-e15d59be1755" (UID: "2620222d-9f11-40c4-b27d-e15d59be1755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:43.082681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.082657    5817 generic.go:356] "Generic (PLEG): container finished" podID="2620222d-9f11-40c4-b27d-e15d59be1755" containerID="682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a" exitCode=0
Nov 03 11:26:43.083161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.083140    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5" event={"ID":"2620222d-9f11-40c4-b27d-e15d59be1755","Type":"ContainerDied","Data":"682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a"}
Nov 03 11:26:43.083351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.083325    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5" event={"ID":"2620222d-9f11-40c4-b27d-e15d59be1755","Type":"ContainerDied","Data":"9b30cec62f95f0da6b4f3b72a446d5730307af803ab89112d80f782f07d672f9"}
Nov 03 11:26:43.083528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.083501    5817 scope.go:122] "RemoveContainer" containerID="682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a"
Nov 03 11:26:43.083941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.083927    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"
Nov 03 11:26:43.128284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.128226    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2620222d-9f11-40c4-b27d-e15d59be1755-kube-api-access-5g5bd" pod "2620222d-9f11-40c4-b27d-e15d59be1755" (UID: "2620222d-9f11-40c4-b27d-e15d59be1755"). InnerVolumeSpecName "kube-api-access-5g5bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:43.130027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.129979    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-maas-api-tls" pod "2620222d-9f11-40c4-b27d-e15d59be1755" (UID: "2620222d-9f11-40c4-b27d-e15d59be1755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:26:43.134896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.134835    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-metrics-certs" pod "2620222d-9f11-40c4-b27d-e15d59be1755" (UID: "2620222d-9f11-40c4-b27d-e15d59be1755"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:26:43.175873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.175833    5817 scope.go:122] "RemoveContainer" containerID="682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a"
Nov 03 11:26:43.176318 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:43.176285    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a\": container with ID starting with 682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a not found: ID does not exist" containerID="682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a"
Nov 03 11:26:43.176449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.176329    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a"} err="failed to get container status \"682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a\": rpc error: code = NotFound desc = could not find container \"682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a\": container with ID starting with 682b53b0693ca45125e0370f3843e3ed069562de86be8ff45abc965af55ab56a not found: ID does not exist"
Nov 03 11:26:43.179767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.179734    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5bd\" (UniqueName: \"kubernetes.io/projected/2620222d-9f11-40c4-b27d-e15d59be1755-kube-api-access-5g5bd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:43.180099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.179772    5817 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:43.180099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.179787    5817 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2620222d-9f11-40c4-b27d-e15d59be1755-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:43.180099 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.179801    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2620222d-9f11-40c4-b27d-e15d59be1755-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:43.278649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.275341    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:43.411720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.411689    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"]
Nov 03 11:26:43.416228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.416202    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9143e1-6f5bc5fcc8-jxvs5"]
Nov 03 11:26:43.431722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.431693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:43.906182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.906148    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2620222d-9f11-40c4-b27d-e15d59be1755" path="/var/lib/kubelet/pods/2620222d-9f11-40c4-b27d-e15d59be1755/volumes"
Nov 03 11:26:43.906793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:43.906774    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dab1729-3db2-46bf-80c8-2ec3b4b57e5c" path="/var/lib/kubelet/pods/7dab1729-3db2-46bf-80c8-2ec3b4b57e5c/volumes"
Nov 03 11:26:44.068593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:44.067705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:44.761596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:44.761410    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:45.513606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:45.512879    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:46.250597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:46.250198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:46.882597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:46.873665    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:47.024667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:47.024635    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"]
Nov 03 11:26:47.029103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:47.029077    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a3e95b-slfjd"]
Nov 03 11:26:47.757600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:47.748638    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:47.903888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:47.903853    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946cee10-f347-4ead-a471-f613ec4f85e9" path="/var/lib/kubelet/pods/946cee10-f347-4ead-a471-f613ec4f85e9/volumes"
Nov 03 11:26:48.448252 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:48.448209    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:49.153050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:49.152877    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:49.332597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:49.330310    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:49.593353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:49.593301    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:49.979599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:49.975997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:50.063848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.063820    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"]
Nov 03 11:26:50.064599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064566    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7dab1729-3db2-46bf-80c8-2ec3b4b57e5c" containerName="revoke-keys"
Nov 03 11:26:50.064654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064601    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dab1729-3db2-46bf-80c8-2ec3b4b57e5c" containerName="revoke-keys"
Nov 03 11:26:50.064654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064633    5817 cpu_manager.go:414] "containerMap: removing container" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy"
Nov 03 11:26:50.064654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064640    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy"
Nov 03 11:26:50.064654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064651    5817 cpu_manager.go:414] "containerMap: removing container" podUID="43d55c2c-2972-4fcf-96ca-0629f294a1b1" containerName="payload-processing"
Nov 03 11:26:50.064808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064658    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d55c2c-2972-4fcf-96ca-0629f294a1b1" containerName="payload-processing"
Nov 03 11:26:50.064808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064671    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2620222d-9f11-40c4-b27d-e15d59be1755" containerName="maas-api"
Nov 03 11:26:50.064808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064677    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2620222d-9f11-40c4-b27d-e15d59be1755" containerName="maas-api"
Nov 03 11:26:50.064808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064689    5817 cpu_manager.go:414] "containerMap: removing container" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy"
Nov 03 11:26:50.064808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064694    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy"
Nov 03 11:26:50.064808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064705    5817 cpu_manager.go:414] "containerMap: removing container" podUID="838aae41-6959-4c5d-b1a8-348ca9375a72" containerName="payload-pre-processing"
Nov 03 11:26:50.064808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064710    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838aae41-6959-4c5d-b1a8-348ca9375a72" containerName="payload-pre-processing"
Nov 03 11:26:50.065065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064851    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dab1729-3db2-46bf-80c8-2ec3b4b57e5c" containerName="revoke-keys"
Nov 03 11:26:50.065065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064866    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy"
Nov 03 11:26:50.065065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064872    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d55c2c-2972-4fcf-96ca-0629f294a1b1" containerName="payload-processing"
Nov 03 11:26:50.065065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064877    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2620222d-9f11-40c4-b27d-e15d59be1755" containerName="maas-api"
Nov 03 11:26:50.065065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064882    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="99eda1c3-5954-45d0-b159-d1f95545bd75" containerName="istio-proxy"
Nov 03 11:26:50.065065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.064889    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="838aae41-6959-4c5d-b1a8-348ca9375a72" containerName="payload-pre-processing"
Nov 03 11:26:50.071767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.070516    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.077348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.077325    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"]
Nov 03 11:26:50.102539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.102518    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:26:50.102750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.102559    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:26:50.162852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.162816    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.163286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.162874    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.163286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.163083    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s46rp\" (UniqueName: \"kubernetes.io/projected/8249260a-7c7b-47b5-8e0b-b5dedff2b952-kube-api-access-s46rp\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.265102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.265015    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s46rp\" (UniqueName: \"kubernetes.io/projected/8249260a-7c7b-47b5-8e0b-b5dedff2b952-kube-api-access-s46rp\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.265275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.265201    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.265275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.265246    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.265869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.265843    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.267789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.267767    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.276521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.276499    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s46rp\" (UniqueName: \"kubernetes.io/projected/8249260a-7c7b-47b5-8e0b-b5dedff2b952-kube-api-access-s46rp\") pod \"maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.390786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.390746    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:50.679632 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:26:50.679601    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8249260a_7c7b_47b5_8e0b_b5dedff2b952.slice/crio-d84eeedf38101439485fa9bd196dd2128d1509e4a94319bd19ed408352c0cf0d WatchSource:0}: Error finding container d84eeedf38101439485fa9bd196dd2128d1509e4a94319bd19ed408352c0cf0d: Status 404 returned error can't find the container with id d84eeedf38101439485fa9bd196dd2128d1509e4a94319bd19ed408352c0cf0d
Nov 03 11:26:50.679806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.679728    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"]
Nov 03 11:26:50.963596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:50.960189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:51.260087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:51.259988    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4" event={"ID":"8249260a-7c7b-47b5-8e0b-b5dedff2b952","Type":"ContainerStarted","Data":"01e52de53752543ee0aa40484822f8b908465051f5b82d7f39bdd0a9f98cd0c3"}
Nov 03 11:26:51.260087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:51.260055    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4" event={"ID":"8249260a-7c7b-47b5-8e0b-b5dedff2b952","Type":"ContainerStarted","Data":"d84eeedf38101439485fa9bd196dd2128d1509e4a94319bd19ed408352c0cf0d"}
Nov 03 11:26:51.279183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:51.279127    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4" podStartSLOduration=1.279108406 podStartE2EDuration="1.279108406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:26:51.276529941 +0000 UTC m=+3441.482112017" watchObservedRunningTime="2026-11-03 11:26:51.279108406 +0000 UTC m=+3441.484690457"
Nov 03 11:26:51.581357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:51.581243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:52.199601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:52.197435    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:52.272141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:52.272108    5817 generic.go:356] "Generic (PLEG): container finished" podID="8249260a-7c7b-47b5-8e0b-b5dedff2b952" containerID="01e52de53752543ee0aa40484822f8b908465051f5b82d7f39bdd0a9f98cd0c3" exitCode=0
Nov 03 11:26:52.272534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:52.272183    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4" event={"ID":"8249260a-7c7b-47b5-8e0b-b5dedff2b952","Type":"ContainerDied","Data":"01e52de53752543ee0aa40484822f8b908465051f5b82d7f39bdd0a9f98cd0c3"}
Nov 03 11:26:52.399802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:52.399751    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9" podUID="be9299d8-3824-4025-a353-119470c37edd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:26:52.415318 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:52.415285    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:26:52.884645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:52.880456    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:53.448749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.448723    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:53.530893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.530844    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-openshift-ca\") pod \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") "
Nov 03 11:26:53.531116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.531097    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-xks-ca\") pod \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") "
Nov 03 11:26:53.569057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.531224    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-openshift-ca" pod "8249260a-7c7b-47b5-8e0b-b5dedff2b952" (UID: "8249260a-7c7b-47b5-8e0b-b5dedff2b952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:53.569057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.531253    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8249260a-7c7b-47b5-8e0b-b5dedff2b952-kube-api-access-s46rp\" (UniqueName: \"kubernetes.io/projected/8249260a-7c7b-47b5-8e0b-b5dedff2b952-kube-api-access-s46rp\") pod \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\" (UID: \"8249260a-7c7b-47b5-8e0b-b5dedff2b952\") "
Nov 03 11:26:53.569057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.531907    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:53.569057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.535386    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8249260a-7c7b-47b5-8e0b-b5dedff2b952-kube-api-access-s46rp" pod "8249260a-7c7b-47b5-8e0b-b5dedff2b952" (UID: "8249260a-7c7b-47b5-8e0b-b5dedff2b952"). InnerVolumeSpecName "kube-api-access-s46rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:53.586613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.578753    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-xks-ca" pod "8249260a-7c7b-47b5-8e0b-b5dedff2b952" (UID: "8249260a-7c7b-47b5-8e0b-b5dedff2b952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:26:53.604691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.604659    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:53.636749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.636717    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s46rp\" (UniqueName: \"kubernetes.io/projected/8249260a-7c7b-47b5-8e0b-b5dedff2b952-kube-api-access-s46rp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:53.636749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:53.636744    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8249260a-7c7b-47b5-8e0b-b5dedff2b952-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.199619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.199467    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:54.272638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.270529    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:26:54.301895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.301860    5817 generic.go:356] "Generic (PLEG): container finished" podID="be9299d8-3824-4025-a353-119470c37edd" containerID="e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65" exitCode=0
Nov 03 11:26:54.302089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.302013    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9" event={"ID":"be9299d8-3824-4025-a353-119470c37edd","Type":"ContainerDied","Data":"e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65"}
Nov 03 11:26:54.302089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.302028    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"
Nov 03 11:26:54.302182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.302092    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9" event={"ID":"be9299d8-3824-4025-a353-119470c37edd","Type":"ContainerDied","Data":"7935094a75fd188ceb6c1a296d9f7467e69b875817d51faebba2eba2cde96aa1"}
Nov 03 11:26:54.302182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.302125    5817 scope.go:122] "RemoveContainer" containerID="e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65"
Nov 03 11:26:54.305187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.304928    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4" event={"ID":"8249260a-7c7b-47b5-8e0b-b5dedff2b952","Type":"ContainerDied","Data":"d84eeedf38101439485fa9bd196dd2128d1509e4a94319bd19ed408352c0cf0d"}
Nov 03 11:26:54.305187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.304979    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d84eeedf38101439485fa9bd196dd2128d1509e4a94319bd19ed408352c0cf0d"
Nov 03 11:26:54.305187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.305001    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"
Nov 03 11:26:54.321761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.321475    5817 scope.go:122] "RemoveContainer" containerID="e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65"
Nov 03 11:26:54.329618 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:26:54.321964    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65\": container with ID starting with e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65 not found: ID does not exist" containerID="e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65"
Nov 03 11:26:54.329618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.322003    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65"} err="failed to get container status \"e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65\": rpc error: code = NotFound desc = could not find container \"e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65\": container with ID starting with e04dbd85e8bab6a0680a560eaf1fadf50fa8ca3040f40877f370f1c60bb9cc65 not found: ID does not exist"
Nov 03 11:26:54.350004 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.349972    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9299d8-3824-4025-a353-119470c37edd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9299d8-3824-4025-a353-119470c37edd-plugins-config-volume\") pod \"be9299d8-3824-4025-a353-119470c37edd\" (UID: \"be9299d8-3824-4025-a353-119470c37edd\") "
Nov 03 11:26:54.350308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.350169    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9299d8-3824-4025-a353-119470c37edd-kube-api-access-j97ng\" (UniqueName: \"kubernetes.io/projected/be9299d8-3824-4025-a353-119470c37edd-kube-api-access-j97ng\") pod \"be9299d8-3824-4025-a353-119470c37edd\" (UID: \"be9299d8-3824-4025-a353-119470c37edd\") "
Nov 03 11:26:54.350467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.350404    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9299d8-3824-4025-a353-119470c37edd-plugins-config-volume" pod "be9299d8-3824-4025-a353-119470c37edd" (UID: "be9299d8-3824-4025-a353-119470c37edd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:54.350895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.350870    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9299d8-3824-4025-a353-119470c37edd-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.353396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.353326    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9299d8-3824-4025-a353-119470c37edd-kube-api-access-j97ng" pod "be9299d8-3824-4025-a353-119470c37edd" (UID: "be9299d8-3824-4025-a353-119470c37edd"). InnerVolumeSpecName "kube-api-access-j97ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:54.452067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.451999    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j97ng\" (UniqueName: \"kubernetes.io/projected/be9299d8-3824-4025-a353-119470c37edd-kube-api-access-j97ng\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.647693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.647658    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"]
Nov 03 11:26:54.661899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.661873    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9143e1-866d986d75-pxkw9"]
Nov 03 11:26:54.720982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.720900    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:54.887109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:54.886792    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:55.499881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:55.499844    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:55.903395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:55.903313    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9299d8-3824-4025-a353-119470c37edd" path="/var/lib/kubelet/pods/be9299d8-3824-4025-a353-119470c37edd/volumes"
Nov 03 11:26:56.091758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:56.091727    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:56.708130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:56.708097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:57.382600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:57.380707    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:58.008699 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:58.006347    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:58.764537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:58.764498    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:26:59.472314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:26:59.471479    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:00.062345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:00.062313    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:00.237390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:00.237357    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"]
Nov 03 11:27:00.336404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:00.336325    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:00.487609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:00.480472    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:00.939248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:00.939214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:01.631300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:01.631099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:02.267959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:02.267920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:02.858240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:02.858209    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:03.043742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:03.043710    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:03.324174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:03.324141    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"]
Nov 03 11:27:03.328595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:03.328547    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9f53bb-cq5j4"]
Nov 03 11:27:03.369420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:03.369383    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:03.518455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:03.518422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:03.873518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:03.873487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:03.905114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:03.905079    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8249260a-7c7b-47b5-8e0b-b5dedff2b952" path="/var/lib/kubelet/pods/8249260a-7c7b-47b5-8e0b-b5dedff2b952/volumes"
Nov 03 11:27:04.122275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:04.122226    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:04.675604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:04.675546    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:04.898362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:04.898333    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:05.238440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:05.238402    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv" podUID="c285b119-1c17-4198-bf9c-07efc70bb500" containerName="payload-pre-processing" containerID="cri-o://6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed" gracePeriod=25
Nov 03 11:27:05.420705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:05.420662    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"]
Nov 03 11:27:05.420924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:05.420901    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy" containerID="cri-o://93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627" gracePeriod=30
Nov 03 11:27:05.504223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:05.504131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:06.240619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:06.238885    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:06.848019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:06.847962    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:06.982295 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:06.982261    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"]
Nov 03 11:27:06.982467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:06.982446    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy" containerID="cri-o://67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285" gracePeriod=30
Nov 03 11:27:07.564380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:07.564330    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:08.268930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:08.268880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:08.630689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:08.630597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:08.883491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:08.883400    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:09.495629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:09.495587    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:10.316445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.307314    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:10.457687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.457659    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:10.590045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.589959    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"]
Nov 03 11:27:10.590502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.590468    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy" containerID="cri-o://59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c" gracePeriod=30
Nov 03 11:27:10.784609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.725423    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:10.940784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.940758    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:27:10.965622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.965523    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-credential-socket\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.965809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.965720    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-token\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-token\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.965897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.965868    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istiod-ca-cert\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.965966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.965910    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-certs\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.965966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.965950    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-envoy\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.966085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.965984    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-data\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.966085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.966035    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-podinfo\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.966085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.966077    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-kube-api-access-m5lpn\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-kube-api-access-m5lpn\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.966261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.966140    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-socket\") pod \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\" (UID: \"9aecee12-bbb1-47ff-86fc-dd034ed4ca05\") "
Nov 03 11:27:10.966441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.966414    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-certs" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:10.966707 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.966683    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-credential-socket" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:10.967010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.966992    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:10.967061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.967017    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:10.967752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.967727    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-socket" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:10.967926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.967903    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-data" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:10.968790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.968733    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istiod-ca-cert" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:10.971532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.969772    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-token" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:10.972104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.972082    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-envoy" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:10.972220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.972140    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-podinfo" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:27:10.972220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:10.972186    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-kube-api-access-m5lpn" pod "9aecee12-bbb1-47ff-86fc-dd034ed4ca05" (UID: "9aecee12-bbb1-47ff-86fc-dd034ed4ca05"). InnerVolumeSpecName "kube-api-access-m5lpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:11.061689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.061644    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:11.068441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.068414    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:11.068638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.068444    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:11.068638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.068458    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:11.068638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.068474    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5lpn\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-kube-api-access-m5lpn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:11.068638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.068488    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:11.068638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.068529    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:11.068638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.068539    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aecee12-bbb1-47ff-86fc-dd034ed4ca05-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:11.148600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.147929    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.155356    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"]
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156530    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8249260a-7c7b-47b5-8e0b-b5dedff2b952" containerName="revoke-keys"
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156550    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8249260a-7c7b-47b5-8e0b-b5dedff2b952" containerName="revoke-keys"
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156634    5817 cpu_manager.go:414] "containerMap: removing container" podUID="be9299d8-3824-4025-a353-119470c37edd" containerName="payload-pre-processing"
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156643    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9299d8-3824-4025-a353-119470c37edd" containerName="payload-pre-processing"
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156670    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy"
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156677    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy"
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156867    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9299d8-3824-4025-a353-119470c37edd" containerName="payload-pre-processing"
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156883    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerName="istio-proxy"
Nov 03 11:27:11.158659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.156908    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8249260a-7c7b-47b5-8e0b-b5dedff2b952" containerName="revoke-keys"
Nov 03 11:27:11.166270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.165896    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"]
Nov 03 11:27:11.166270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.165980    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.172509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.172483    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.172803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.172779    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5z29\" (UniqueName: \"kubernetes.io/projected/086be46d-9ee1-4de8-b6fd-76ce9e309c13-kube-api-access-l5z29\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.174245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.173322    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.196426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.193530    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:27:11.201930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.201903    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:27:11.276688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.276643    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.276822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.276724    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.276822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.276771    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5z29\" (UniqueName: \"kubernetes.io/projected/086be46d-9ee1-4de8-b6fd-76ce9e309c13-kube-api-access-l5z29\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.277376 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.277357    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.279154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.279135    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.285191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.285169    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5z29\" (UniqueName: \"kubernetes.io/projected/086be46d-9ee1-4de8-b6fd-76ce9e309c13-kube-api-access-l5z29\") pod \"maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.493621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.493584    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:11.497785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.497756    5817 generic.go:356] "Generic (PLEG): container finished" podID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" containerID="93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627" exitCode=0
Nov 03 11:27:11.497938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.497797    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" event={"ID":"9aecee12-bbb1-47ff-86fc-dd034ed4ca05","Type":"ContainerDied","Data":"93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627"}
Nov 03 11:27:11.497938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.497811    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"
Nov 03 11:27:11.497938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.497871    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps" event={"ID":"9aecee12-bbb1-47ff-86fc-dd034ed4ca05","Type":"ContainerDied","Data":"09374aa6220b16bb566557c2a4d4b5ffbf590687a2f85bc30834cd2b726d89d6"}
Nov 03 11:27:11.497938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.497911    5817 scope.go:122] "RemoveContainer" containerID="93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627"
Nov 03 11:27:11.518704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.518681    5817 scope.go:122] "RemoveContainer" containerID="93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627"
Nov 03 11:27:11.519128 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:27:11.519099    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627\": container with ID starting with 93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627 not found: ID does not exist" containerID="93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627"
Nov 03 11:27:11.519247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.519137    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627"} err="failed to get container status \"93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627\": rpc error: code = NotFound desc = could not find container \"93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627\": container with ID starting with 93065b767caab5bc10b879593895c6e1ce0f63d4bdba662f78d57a9de53cf627 not found: ID does not exist"
Nov 03 11:27:11.557079 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.557027    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"]
Nov 03 11:27:11.563921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.563895    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-n55ps"]
Nov 03 11:27:11.807697 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:27:11.807670    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086be46d_9ee1_4de8_b6fd_76ce9e309c13.slice/crio-5c5c636bd2147de549643c2cee715fa99ef41ce4598386c3d9a9b2aa2dd201d0 WatchSource:0}: Error finding container 5c5c636bd2147de549643c2cee715fa99ef41ce4598386c3d9a9b2aa2dd201d0: Status 404 returned error can't find the container with id 5c5c636bd2147de549643c2cee715fa99ef41ce4598386c3d9a9b2aa2dd201d0
Nov 03 11:27:11.807853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.807830    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"]
Nov 03 11:27:11.904272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.904244    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aecee12-bbb1-47ff-86fc-dd034ed4ca05" path="/var/lib/kubelet/pods/9aecee12-bbb1-47ff-86fc-dd034ed4ca05/volumes"
Nov 03 11:27:11.962601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:11.953679    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:12.261115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.261089    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:27:12.292844 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.292818    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-credential-socket\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.292844 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.292849    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04aa18f3-4823-4266-9257-f43ae8d69a7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aa18f3-4823-4266-9257-f43ae8d69a7d-istiod-ca-cert\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.293086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.292870    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-podinfo\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.293086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.292946    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-data\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.293086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293004    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-kube-api-access-5mmf6\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-kube-api-access-5mmf6\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.293086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293036    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-socket\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.293331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293108    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-credential-socket" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:12.293331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293147    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-envoy\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.293331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293217    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-token\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-token\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.293331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293277    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-data" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:12.293331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293283    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-certs\") pod \"04aa18f3-4823-4266-9257-f43ae8d69a7d\" (UID: \"04aa18f3-4823-4266-9257-f43ae8d69a7d\") "
Nov 03 11:27:12.293629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293455    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-certs" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:12.293697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.293634    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04aa18f3-4823-4266-9257-f43ae8d69a7d-istiod-ca-cert" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:12.294250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.294221    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.294401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.294253    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.294401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.294270    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aa18f3-4823-4266-9257-f43ae8d69a7d-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.294401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.294285    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.294591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.294467    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-socket" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:12.296501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.296478    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-envoy" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:12.296816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.296795    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-podinfo" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:27:12.297027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.297006    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-kube-api-access-5mmf6" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "kube-api-access-5mmf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:12.298029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.298005    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-token" pod "04aa18f3-4823-4266-9257-f43ae8d69a7d" (UID: "04aa18f3-4823-4266-9257-f43ae8d69a7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:12.400051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.399929    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mmf6\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-kube-api-access-5mmf6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.400051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.399996    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.400051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.400013    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.400051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.400026    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.400051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.400038    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aa18f3-4823-4266-9257-f43ae8d69a7d-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:12.508938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.508908    5817 generic.go:356] "Generic (PLEG): container finished" podID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerID="67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285" exitCode=0
Nov 03 11:27:12.509388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.508964    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"
Nov 03 11:27:12.509388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.508991    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" event={"ID":"04aa18f3-4823-4266-9257-f43ae8d69a7d","Type":"ContainerDied","Data":"67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285"}
Nov 03 11:27:12.509388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.509042    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8" event={"ID":"04aa18f3-4823-4266-9257-f43ae8d69a7d","Type":"ContainerDied","Data":"937f907da405fce3aae2ef2bc4d80cff7b6d5ed58ec863c77e17955e947d564d"}
Nov 03 11:27:12.509388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.509055    5817 scope.go:122] "RemoveContainer" containerID="67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285"
Nov 03 11:27:12.512403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.512382    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq" event={"ID":"086be46d-9ee1-4de8-b6fd-76ce9e309c13","Type":"ContainerStarted","Data":"876e631a8f801aec0a26b7bab3f5fb72cb110890ac40630a9f97b6c257dbe9a6"}
Nov 03 11:27:12.512519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.512420    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq" event={"ID":"086be46d-9ee1-4de8-b6fd-76ce9e309c13","Type":"ContainerStarted","Data":"5c5c636bd2147de549643c2cee715fa99ef41ce4598386c3d9a9b2aa2dd201d0"}
Nov 03 11:27:12.521930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.521912    5817 scope.go:122] "RemoveContainer" containerID="67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285"
Nov 03 11:27:12.522222 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:27:12.522200    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285\": container with ID starting with 67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285 not found: ID does not exist" containerID="67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285"
Nov 03 11:27:12.522302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.522228    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285"} err="failed to get container status \"67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285\": rpc error: code = NotFound desc = could not find container \"67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285\": container with ID starting with 67dd7615ac24bddaaa827be13917f76df00919239e355e72e7725f154947d285 not found: ID does not exist"
Nov 03 11:27:12.538665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.538618    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq" podStartSLOduration=1.53860051 podStartE2EDuration="1.53860051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:27:12.533216719 +0000 UTC m=+3462.738798770" watchObservedRunningTime="2026-11-03 11:27:12.53860051 +0000 UTC m=+3462.744182565"
Nov 03 11:27:12.549311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.549278    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"]
Nov 03 11:27:12.555773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.555750    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mbnr8"]
Nov 03 11:27:12.872595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:12.872221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:13.523410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:13.523374    5817 generic.go:356] "Generic (PLEG): container finished" podID="086be46d-9ee1-4de8-b6fd-76ce9e309c13" containerID="876e631a8f801aec0a26b7bab3f5fb72cb110890ac40630a9f97b6c257dbe9a6" exitCode=0
Nov 03 11:27:13.523834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:13.523457    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq" event={"ID":"086be46d-9ee1-4de8-b6fd-76ce9e309c13","Type":"ContainerDied","Data":"876e631a8f801aec0a26b7bab3f5fb72cb110890ac40630a9f97b6c257dbe9a6"}
Nov 03 11:27:13.671614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:13.671154    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:13.904610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:13.904502    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" path="/var/lib/kubelet/pods/04aa18f3-4823-4266-9257-f43ae8d69a7d/volumes"
Nov 03 11:27:14.394612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.393385    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:14.699415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.699390    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:14.738045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.738010    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-xks-ca\") pod \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") "
Nov 03 11:27:14.738268 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.738102    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-openshift-ca\") pod \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") "
Nov 03 11:27:14.738495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.738451    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-openshift-ca" pod "086be46d-9ee1-4de8-b6fd-76ce9e309c13" (UID: "086be46d-9ee1-4de8-b6fd-76ce9e309c13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:14.738665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.738535    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086be46d-9ee1-4de8-b6fd-76ce9e309c13-kube-api-access-l5z29\" (UniqueName: \"kubernetes.io/projected/086be46d-9ee1-4de8-b6fd-76ce9e309c13-kube-api-access-l5z29\") pod \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\" (UID: \"086be46d-9ee1-4de8-b6fd-76ce9e309c13\") "
Nov 03 11:27:14.739440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.739416    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:14.741920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.741896    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086be46d-9ee1-4de8-b6fd-76ce9e309c13-kube-api-access-l5z29" pod "086be46d-9ee1-4de8-b6fd-76ce9e309c13" (UID: "086be46d-9ee1-4de8-b6fd-76ce9e309c13"). InnerVolumeSpecName "kube-api-access-l5z29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:14.742101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.742083    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-xks-ca" pod "086be46d-9ee1-4de8-b6fd-76ce9e309c13" (UID: "086be46d-9ee1-4de8-b6fd-76ce9e309c13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:27:14.840452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.840422    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086be46d-9ee1-4de8-b6fd-76ce9e309c13-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:14.840452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:14.840450    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5z29\" (UniqueName: \"kubernetes.io/projected/086be46d-9ee1-4de8-b6fd-76ce9e309c13-kube-api-access-l5z29\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:15.032473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:15.032430    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv" podUID="c285b119-1c17-4198-bf9c-07efc70bb500" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:15.129111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:15.129077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:15.545171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:15.545144    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"
Nov 03 11:27:15.545366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:15.545183    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq" event={"ID":"086be46d-9ee1-4de8-b6fd-76ce9e309c13","Type":"ContainerDied","Data":"5c5c636bd2147de549643c2cee715fa99ef41ce4598386c3d9a9b2aa2dd201d0"}
Nov 03 11:27:15.545366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:15.545228    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5c636bd2147de549643c2cee715fa99ef41ce4598386c3d9a9b2aa2dd201d0"
Nov 03 11:27:15.875646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:15.873392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:15.952946 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:15.952919    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:27:16.104871 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.104840    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-token\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-token\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.104921    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-envoy\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.104955    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istiod-ca-cert\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.104984    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-podinfo\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105076    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-data\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105112    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-certs\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105160    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-credential-socket\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105216    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-kube-api-access-x49dt\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-kube-api-access-x49dt\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105292    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-socket\") pod \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\" (UID: \"129748c3-369d-4a33-9cda-7f5ebabf7d7e\") "
Nov 03 11:27:16.105435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105373    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istiod-ca-cert" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:16.105435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105413    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-data" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:16.105720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105695    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-credential-socket" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:16.105941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105889    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-socket" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:16.106233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.105982    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-certs" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:16.106365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.106188    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.106560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.106371    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.106560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.106407    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.107634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.107608    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-podinfo" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:27:16.108048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.108022    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-token" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:16.108152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.108063    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-kube-api-access-x49dt" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "kube-api-access-x49dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:16.108281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.108264    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-envoy" pod "129748c3-369d-4a33-9cda-7f5ebabf7d7e" (UID: "129748c3-369d-4a33-9cda-7f5ebabf7d7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:16.207950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.207921    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.207950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.207949    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.208121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.207958    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.208121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.207974    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129748c3-369d-4a33-9cda-7f5ebabf7d7e-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.208121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.207984    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129748c3-369d-4a33-9cda-7f5ebabf7d7e-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.208121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.207992    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x49dt\" (UniqueName: \"kubernetes.io/projected/129748c3-369d-4a33-9cda-7f5ebabf7d7e-kube-api-access-x49dt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:16.594651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.591490    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:16.599402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.599359    5817 generic.go:356] "Generic (PLEG): container finished" podID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerID="59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c" exitCode=0
Nov 03 11:27:16.599648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.599416    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" event={"ID":"129748c3-369d-4a33-9cda-7f5ebabf7d7e","Type":"ContainerDied","Data":"59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c"}
Nov 03 11:27:16.599648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.599516    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs" event={"ID":"129748c3-369d-4a33-9cda-7f5ebabf7d7e","Type":"ContainerDied","Data":"c666034d1cb691c0ad2354be613b3072a57303c4aceab0431278241bc21d1f1b"}
Nov 03 11:27:16.599648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.599550    5817 scope.go:122] "RemoveContainer" containerID="59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c"
Nov 03 11:27:16.599884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.599767    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"
Nov 03 11:27:16.628924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.628894    5817 scope.go:122] "RemoveContainer" containerID="878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e"
Nov 03 11:27:16.642932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.642880    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"]
Nov 03 11:27:16.643709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.643689    5817 scope.go:122] "RemoveContainer" containerID="59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c"
Nov 03 11:27:16.644935 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:27:16.644911    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c\": container with ID starting with 59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c not found: ID does not exist" containerID="59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c"
Nov 03 11:27:16.645051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.644946    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c"} err="failed to get container status \"59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c\": rpc error: code = NotFound desc = could not find container \"59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c\": container with ID starting with 59a05837e7f53a26b6f05d500dcc994fb5c5739c09d9c4a4c844960f6801302c not found: ID does not exist"
Nov 03 11:27:16.645051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.644992    5817 scope.go:122] "RemoveContainer" containerID="878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e"
Nov 03 11:27:16.645362 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:27:16.645342    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e\": container with ID starting with 878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e not found: ID does not exist" containerID="878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e"
Nov 03 11:27:16.645429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.645370    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e"} err="failed to get container status \"878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e\": rpc error: code = NotFound desc = could not find container \"878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e\": container with ID starting with 878be5b159b8bafba3651a8ab33b2d913c3e0005dc460cde7f2034a41617689e not found: ID does not exist"
Nov 03 11:27:16.649680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:16.649658    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-snqfs"]
Nov 03 11:27:17.280597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:17.278764    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:17.903196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:17.903158    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" path="/var/lib/kubelet/pods/129748c3-369d-4a33-9cda-7f5ebabf7d7e/volumes"
Nov 03 11:27:17.987216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:17.987182    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:18.600600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:18.597592    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:18.965975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:18.965941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:19.289692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:19.285353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:19.936983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:19.936947    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:20.568046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:20.563762    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:21.199802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:21.199765    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"]
Nov 03 11:27:21.284545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:21.284496    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:21.437769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:21.437734    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:22.095885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:22.095843    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:22.413961 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:27:22.413872    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:27:22.880182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:22.879911    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:23.745640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:23.745610    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:24.464667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:24.464630    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:25.031488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.031446    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv" podUID="c285b119-1c17-4198-bf9c-07efc70bb500" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:25.086221 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.086183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:25.548058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.548025    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:27:25.682781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.682750    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c285b119-1c17-4198-bf9c-07efc70bb500-kube-api-access-jg9cp\" (UniqueName: \"kubernetes.io/projected/c285b119-1c17-4198-bf9c-07efc70bb500-kube-api-access-jg9cp\") pod \"c285b119-1c17-4198-bf9c-07efc70bb500\" (UID: \"c285b119-1c17-4198-bf9c-07efc70bb500\") "
Nov 03 11:27:25.683010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.682986    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c285b119-1c17-4198-bf9c-07efc70bb500-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c285b119-1c17-4198-bf9c-07efc70bb500-plugins-config-volume\") pod \"c285b119-1c17-4198-bf9c-07efc70bb500\" (UID: \"c285b119-1c17-4198-bf9c-07efc70bb500\") "
Nov 03 11:27:25.683472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.683434    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c285b119-1c17-4198-bf9c-07efc70bb500-plugins-config-volume" pod "c285b119-1c17-4198-bf9c-07efc70bb500" (UID: "c285b119-1c17-4198-bf9c-07efc70bb500"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:25.683683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.683661    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c285b119-1c17-4198-bf9c-07efc70bb500-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:25.685644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.685617    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c285b119-1c17-4198-bf9c-07efc70bb500-kube-api-access-jg9cp" pod "c285b119-1c17-4198-bf9c-07efc70bb500" (UID: "c285b119-1c17-4198-bf9c-07efc70bb500"). InnerVolumeSpecName "kube-api-access-jg9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:25.759127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.759096    5817 generic.go:356] "Generic (PLEG): container finished" podID="c285b119-1c17-4198-bf9c-07efc70bb500" containerID="6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed" exitCode=0
Nov 03 11:27:25.759307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.759138    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"
Nov 03 11:27:25.759307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.759184    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv" event={"ID":"c285b119-1c17-4198-bf9c-07efc70bb500","Type":"ContainerDied","Data":"6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed"}
Nov 03 11:27:25.759307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.759241    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv" event={"ID":"c285b119-1c17-4198-bf9c-07efc70bb500","Type":"ContainerDied","Data":"eec5aa12b1ef553cbf9f36a970f09526ccfec906bb595f1ffc379df682e6dd4a"}
Nov 03 11:27:25.759307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.759270    5817 scope.go:122] "RemoveContainer" containerID="6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed"
Nov 03 11:27:25.818626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.815280    5817 scope.go:122] "RemoveContainer" containerID="6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed"
Nov 03 11:27:25.825941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.825904    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"]
Nov 03 11:27:25.828132 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:27:25.828089    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed\": container with ID starting with 6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed not found: ID does not exist" containerID="6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed"
Nov 03 11:27:25.829388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.828346    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed"} err="failed to get container status \"6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed\": rpc error: code = NotFound desc = could not find container \"6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed\": container with ID starting with 6699c08a8bfa2ae86e3adbb5e8be992381aba42f2671593eb9c9e96cdbe3f5ed not found: ID does not exist"
Nov 03 11:27:25.830014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.829990    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:25.832515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.832492    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg9cp\" (UniqueName: \"kubernetes.io/projected/c285b119-1c17-4198-bf9c-07efc70bb500-kube-api-access-jg9cp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:25.839631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.837539    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9f53bb-57f76df9cf-8bglv"]
Nov 03 11:27:25.906748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:25.906713    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c285b119-1c17-4198-bf9c-07efc70bb500" path="/var/lib/kubelet/pods/c285b119-1c17-4198-bf9c-07efc70bb500/volumes"
Nov 03 11:27:26.200669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:26.200625    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f" podUID="aec2a5fd-6fc8-4f67-982f-c2cc457bc645" containerName="payload-processing" containerID="cri-o://8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab" gracePeriod=25
Nov 03 11:27:26.496651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:26.490605    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:27.205353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:27.205222    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:27.825593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:27.825130    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:28.474352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:28.474314    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:29.179629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:29.175500    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:29.662332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:29.662248    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:29.760445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:29.760411    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"]
Nov 03 11:27:29.764371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:29.764349    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d9f2ce-rxzlq"]
Nov 03 11:27:29.796014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:29.795980    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:29.909412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:29.909376    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086be46d-9ee1-4de8-b6fd-76ce9e309c13" path="/var/lib/kubelet/pods/086be46d-9ee1-4de8-b6fd-76ce9e309c13/volumes"
Nov 03 11:27:29.953603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:29.952138    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:30.323972 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:30.323880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:30.923894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:30.923860    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:31.785616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:31.775949    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:32.457206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:32.457174    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:33.112392 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:33.111853    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:33.784939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:33.784864    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:34.708013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:34.707979    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:35.344006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:35.343960    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:35.574545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:35.574514    5817 scope.go:122] "RemoveContainer" containerID="43dec831b8aabfdcbc3741cf7c1fc407d8d993e260732c6c680bd7aed6f26175"
Nov 03 11:27:35.586863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:35.586843    5817 scope.go:122] "RemoveContainer" containerID="f7c755e84aa0d00fa6353b919f5ac6d1be8dcf9c4dc82273b38f57963e52a83a"
Nov 03 11:27:35.597148 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:35.597128    5817 scope.go:122] "RemoveContainer" containerID="b1215fb51ef829e7120d10b0d7f1b76547d9cd7fdddde475a40ea763c7b576b5"
Nov 03 11:27:35.607859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:35.607839    5817 scope.go:122] "RemoveContainer" containerID="83102fa181170c0e8c3e9cf946bdb00792ae0e62987bbb017142e647eefd6dfe"
Nov 03 11:27:35.976401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:35.976369    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f" podUID="aec2a5fd-6fc8-4f67-982f-c2cc457bc645" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:36.211723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:36.211688    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:37.039670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:37.035238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:37.064156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:37.064120    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"]
Nov 03 11:27:37.070415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:37.070384    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce03bd71-v2vcr"]
Nov 03 11:27:37.697655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:37.697618    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:37.860079 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:37.860036    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:37.904687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:37.904641    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa059d4c-5a2f-461d-a414-3ca128998417" path="/var/lib/kubelet/pods/aa059d4c-5a2f-461d-a414-3ca128998417/volumes"
Nov 03 11:27:38.033641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.032157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:38.533604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.533552    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"]
Nov 03 11:27:38.534695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534676    5817 cpu_manager.go:414] "containerMap: removing container" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy"
Nov 03 11:27:38.534798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534698    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy"
Nov 03 11:27:38.534798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534734    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c285b119-1c17-4198-bf9c-07efc70bb500" containerName="payload-pre-processing"
Nov 03 11:27:38.534798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534743    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c285b119-1c17-4198-bf9c-07efc70bb500" containerName="payload-pre-processing"
Nov 03 11:27:38.534798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534763    5817 cpu_manager.go:414] "containerMap: removing container" podUID="086be46d-9ee1-4de8-b6fd-76ce9e309c13" containerName="revoke-keys"
Nov 03 11:27:38.534798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534770    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086be46d-9ee1-4de8-b6fd-76ce9e309c13" containerName="revoke-keys"
Nov 03 11:27:38.534798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534789    5817 cpu_manager.go:414] "containerMap: removing container" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy"
Nov 03 11:27:38.534798 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534797    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy"
Nov 03 11:27:38.535039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534961    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy"
Nov 03 11:27:38.535039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534971    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="04aa18f3-4823-4266-9257-f43ae8d69a7d" containerName="istio-proxy"
Nov 03 11:27:38.535039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.534991    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy"
Nov 03 11:27:38.535039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.535004    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c285b119-1c17-4198-bf9c-07efc70bb500" containerName="payload-pre-processing"
Nov 03 11:27:38.535039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.535015    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="086be46d-9ee1-4de8-b6fd-76ce9e309c13" containerName="revoke-keys"
Nov 03 11:27:38.541176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.540922    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.547705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.547679    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"]
Nov 03 11:27:38.576926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.576905    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:27:38.577091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.577075    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:27:38.622451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.622320    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:38.701371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.701341    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lcl\" (UniqueName: \"kubernetes.io/projected/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-kube-api-access-67lcl\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.701638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.701493    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.701638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.701523    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.803168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.803078    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.803168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.803116    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.803427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.803273    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lcl\" (UniqueName: \"kubernetes.io/projected/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-kube-api-access-67lcl\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.804011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.803983    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.805808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.805781    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.811940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.811913    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lcl\" (UniqueName: \"kubernetes.io/projected/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-kube-api-access-67lcl\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:38.860989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:38.860966    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:39.144310 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:39.144281    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"]
Nov 03 11:27:39.144464 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:27:39.144395    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d6fab4_c76a_42b2_9aec_c7e46dcc5531.slice/crio-00bfec22c2f792124c8cf882e4e101664aa60ec6d7d3790f06b06917e34379d9 WatchSource:0}: Error finding container 00bfec22c2f792124c8cf882e4e101664aa60ec6d7d3790f06b06917e34379d9: Status 404 returned error can't find the container with id 00bfec22c2f792124c8cf882e4e101664aa60ec6d7d3790f06b06917e34379d9
Nov 03 11:27:39.432705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:39.431920    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:39.892253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:39.892168    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff" event={"ID":"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531","Type":"ContainerStarted","Data":"af2e028ceeec37a2091038669e3e2b1a53588df3cf196fbfc81fce881a0b8ab2"}
Nov 03 11:27:39.892253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:39.892227    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff" event={"ID":"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531","Type":"ContainerStarted","Data":"00bfec22c2f792124c8cf882e4e101664aa60ec6d7d3790f06b06917e34379d9"}
Nov 03 11:27:39.920809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:39.920758    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff" podStartSLOduration=1.920742265 podStartE2EDuration="1.920742265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:27:39.917019142 +0000 UTC m=+3490.122601209" watchObservedRunningTime="2026-11-03 11:27:39.920742265 +0000 UTC m=+3490.126324316"
Nov 03 11:27:40.173106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:40.173018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:40.906831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:40.906791    5817 generic.go:356] "Generic (PLEG): container finished" podID="b2d6fab4-c76a-42b2-9aec-c7e46dcc5531" containerID="af2e028ceeec37a2091038669e3e2b1a53588df3cf196fbfc81fce881a0b8ab2" exitCode=0
Nov 03 11:27:40.907339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:40.906873    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff" event={"ID":"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531","Type":"ContainerDied","Data":"af2e028ceeec37a2091038669e3e2b1a53588df3cf196fbfc81fce881a0b8ab2"}
Nov 03 11:27:40.915486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:40.915456    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:41.990184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:41.990146    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:42.147980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.147945    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:42.164698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.163434    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-xks-ca\") pod \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") "
Nov 03 11:27:42.164698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.163498    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-kube-api-access-67lcl\" (UniqueName: \"kubernetes.io/projected/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-kube-api-access-67lcl\") pod \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") "
Nov 03 11:27:42.164698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.163775    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-openshift-ca\") pod \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\" (UID: \"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531\") "
Nov 03 11:27:42.165075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.165036    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-openshift-ca" pod "b2d6fab4-c76a-42b2-9aec-c7e46dcc5531" (UID: "b2d6fab4-c76a-42b2-9aec-c7e46dcc5531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:42.167080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.167053    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-xks-ca" pod "b2d6fab4-c76a-42b2-9aec-c7e46dcc5531" (UID: "b2d6fab4-c76a-42b2-9aec-c7e46dcc5531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:27:42.168120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.168074    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-kube-api-access-67lcl" pod "b2d6fab4-c76a-42b2-9aec-c7e46dcc5531" (UID: "b2d6fab4-c76a-42b2-9aec-c7e46dcc5531"). InnerVolumeSpecName "kube-api-access-67lcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:42.266470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.266386    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:42.266470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.266421    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:42.266470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.266435    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lcl\" (UniqueName: \"kubernetes.io/projected/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531-kube-api-access-67lcl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:42.493115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.492198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:42.976301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.976265    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"
Nov 03 11:27:42.976542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.976310    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff" event={"ID":"b2d6fab4-c76a-42b2-9aec-c7e46dcc5531","Type":"ContainerDied","Data":"00bfec22c2f792124c8cf882e4e101664aa60ec6d7d3790f06b06917e34379d9"}
Nov 03 11:27:42.976542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:42.976359    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00bfec22c2f792124c8cf882e4e101664aa60ec6d7d3790f06b06917e34379d9"
Nov 03 11:27:43.449064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:43.449034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:44.215249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:44.215165    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:44.939142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:44.939096    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:45.835296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:45.835256    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:45.975818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:45.975776    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f" podUID="aec2a5fd-6fc8-4f67-982f-c2cc457bc645" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:46.253837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.253795    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"]
Nov 03 11:27:46.254494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.254478    5817 cpu_manager.go:414] "containerMap: removing container" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy"
Nov 03 11:27:46.254494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.254493    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129748c3-369d-4a33-9cda-7f5ebabf7d7e" containerName="istio-proxy"
Nov 03 11:27:46.254681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.254508    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b2d6fab4-c76a-42b2-9aec-c7e46dcc5531" containerName="revoke-keys"
Nov 03 11:27:46.254681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.254514    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d6fab4-c76a-42b2-9aec-c7e46dcc5531" containerName="revoke-keys"
Nov 03 11:27:46.254681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.254657    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d6fab4-c76a-42b2-9aec-c7e46dcc5531" containerName="revoke-keys"
Nov 03 11:27:46.258267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.258250    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.266845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.266820    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:27:46.267509 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.267486    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"]
Nov 03 11:27:46.268153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.268131    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:27:46.352192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.352161    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.352323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.352291    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.352407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.352388    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fv8\" (UniqueName: \"kubernetes.io/projected/7253e0f4-622a-47f7-afd3-c95f43a25420-kube-api-access-69fv8\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.454971 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.454888    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.456006 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.455934    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fv8\" (UniqueName: \"kubernetes.io/projected/7253e0f4-622a-47f7-afd3-c95f43a25420-kube-api-access-69fv8\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.456882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.456457    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.457290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.457261    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.460093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.460068    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.469445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.469406    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fv8\" (UniqueName: \"kubernetes.io/projected/7253e0f4-622a-47f7-afd3-c95f43a25420-kube-api-access-69fv8\") pod \"maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.494378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.494355    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:27:46.557565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.557531    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-kube-api-access-vjghv\" (UniqueName: \"kubernetes.io/projected/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-kube-api-access-vjghv\") pod \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\" (UID: \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\") "
Nov 03 11:27:46.557942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.557922    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-plugins-config-volume\") pod \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\" (UID: \"aec2a5fd-6fc8-4f67-982f-c2cc457bc645\") "
Nov 03 11:27:46.558434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.558394    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-plugins-config-volume" pod "aec2a5fd-6fc8-4f67-982f-c2cc457bc645" (UID: "aec2a5fd-6fc8-4f67-982f-c2cc457bc645"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:46.558879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.558833    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:46.560442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.560414    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-kube-api-access-vjghv" pod "aec2a5fd-6fc8-4f67-982f-c2cc457bc645" (UID: "aec2a5fd-6fc8-4f67-982f-c2cc457bc645"). InnerVolumeSpecName "kube-api-access-vjghv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:46.585419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.585395    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:46.660853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.660815    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjghv\" (UniqueName: \"kubernetes.io/projected/aec2a5fd-6fc8-4f67-982f-c2cc457bc645-kube-api-access-vjghv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:46.878561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:46.878538    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"]
Nov 03 11:27:46.880137 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:27:46.880107    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7253e0f4_622a_47f7_afd3_c95f43a25420.slice/crio-007e93538436671a18ee4c2831defc7f1dbc33cf8effa92486b82983a2959d08 WatchSource:0}: Error finding container 007e93538436671a18ee4c2831defc7f1dbc33cf8effa92486b82983a2959d08: Status 404 returned error can't find the container with id 007e93538436671a18ee4c2831defc7f1dbc33cf8effa92486b82983a2959d08
Nov 03 11:27:47.042322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.042286    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8" event={"ID":"7253e0f4-622a-47f7-afd3-c95f43a25420","Type":"ContainerStarted","Data":"007e93538436671a18ee4c2831defc7f1dbc33cf8effa92486b82983a2959d08"}
Nov 03 11:27:47.044373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.044315    5817 generic.go:356] "Generic (PLEG): container finished" podID="aec2a5fd-6fc8-4f67-982f-c2cc457bc645" containerID="8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab" exitCode=0
Nov 03 11:27:47.044373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.044357    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f" event={"ID":"aec2a5fd-6fc8-4f67-982f-c2cc457bc645","Type":"ContainerDied","Data":"8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab"}
Nov 03 11:27:47.044611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.044398    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f" event={"ID":"aec2a5fd-6fc8-4f67-982f-c2cc457bc645","Type":"ContainerDied","Data":"8a767246a9ab6643c3d134c6f9e64df4cfbbb455e13ade5dd85f8fd769843884"}
Nov 03 11:27:47.044611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.044427    5817 scope.go:122] "RemoveContainer" containerID="8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab"
Nov 03 11:27:47.044749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.044614    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"
Nov 03 11:27:47.075531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.075507    5817 scope.go:122] "RemoveContainer" containerID="8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab"
Nov 03 11:27:47.075885 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:27:47.075865    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab\": container with ID starting with 8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab not found: ID does not exist" containerID="8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab"
Nov 03 11:27:47.075950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.075894    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab"} err="failed to get container status \"8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab\": rpc error: code = NotFound desc = could not find container \"8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab\": container with ID starting with 8ab86ae3a6312374dc63b6d00bff4ea50805b796028acf9ed54350fa556bcbab not found: ID does not exist"
Nov 03 11:27:47.101553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.101529    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"]
Nov 03 11:27:47.112248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.112215    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:47.114741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.114721    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d9f2ce-66f5865b5f-c5s4f"]
Nov 03 11:27:47.905544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:47.905513    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec2a5fd-6fc8-4f67-982f-c2cc457bc645" path="/var/lib/kubelet/pods/aec2a5fd-6fc8-4f67-982f-c2cc457bc645/volumes"
Nov 03 11:27:48.028540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:48.028513    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"]
Nov 03 11:27:48.032620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:48.032596    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b326624-s8dnd"]
Nov 03 11:27:48.089147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:48.088326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:48.117534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:48.117504    5817 generic.go:356] "Generic (PLEG): container finished" podID="7253e0f4-622a-47f7-afd3-c95f43a25420" containerID="b18df56970b8d5d5e48f80a2b343689598140a0ebc3a5def68fd7c618fb378d5" exitCode=0
Nov 03 11:27:48.117767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:48.117599    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8" event={"ID":"7253e0f4-622a-47f7-afd3-c95f43a25420","Type":"ContainerDied","Data":"b18df56970b8d5d5e48f80a2b343689598140a0ebc3a5def68fd7c618fb378d5"}
Nov 03 11:27:48.827513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:48.827473    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:49.054769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.054732    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:49.328497 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.328447    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:49.429193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.429158    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-xks-ca\") pod \"7253e0f4-622a-47f7-afd3-c95f43a25420\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") "
Nov 03 11:27:49.429388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.429376    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-openshift-ca\") pod \"7253e0f4-622a-47f7-afd3-c95f43a25420\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") "
Nov 03 11:27:49.429513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.429458    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7253e0f4-622a-47f7-afd3-c95f43a25420-kube-api-access-69fv8\" (UniqueName: \"kubernetes.io/projected/7253e0f4-622a-47f7-afd3-c95f43a25420-kube-api-access-69fv8\") pod \"7253e0f4-622a-47f7-afd3-c95f43a25420\" (UID: \"7253e0f4-622a-47f7-afd3-c95f43a25420\") "
Nov 03 11:27:49.429777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.429753    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-openshift-ca" pod "7253e0f4-622a-47f7-afd3-c95f43a25420" (UID: "7253e0f4-622a-47f7-afd3-c95f43a25420"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:49.430191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.430126    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:49.431473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.431449    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-xks-ca" pod "7253e0f4-622a-47f7-afd3-c95f43a25420" (UID: "7253e0f4-622a-47f7-afd3-c95f43a25420"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:27:49.431863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.431836    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7253e0f4-622a-47f7-afd3-c95f43a25420-kube-api-access-69fv8" pod "7253e0f4-622a-47f7-afd3-c95f43a25420" (UID: "7253e0f4-622a-47f7-afd3-c95f43a25420"). InnerVolumeSpecName "kube-api-access-69fv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:49.531349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.531313    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69fv8\" (UniqueName: \"kubernetes.io/projected/7253e0f4-622a-47f7-afd3-c95f43a25420-kube-api-access-69fv8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:49.531349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.531343    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7253e0f4-622a-47f7-afd3-c95f43a25420-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:49.771386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.769805    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:49.902904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:49.902873    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629e49ef-8016-461e-9eba-7ab15bab0e31" path="/var/lib/kubelet/pods/629e49ef-8016-461e-9eba-7ab15bab0e31/volumes"
Nov 03 11:27:50.152113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:50.152030    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"
Nov 03 11:27:50.152113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:50.152046    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8" event={"ID":"7253e0f4-622a-47f7-afd3-c95f43a25420","Type":"ContainerDied","Data":"007e93538436671a18ee4c2831defc7f1dbc33cf8effa92486b82983a2959d08"}
Nov 03 11:27:50.152113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:50.152098    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="007e93538436671a18ee4c2831defc7f1dbc33cf8effa92486b82983a2959d08"
Nov 03 11:27:50.529768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:50.529736    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:51.404890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:51.404844    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:52.275595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:52.269182    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:52.417182 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:27:52.417146    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:27:52.863094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:52.863038    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:53.053636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:53.053478    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:53.164325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:53.164240    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"]
Nov 03 11:27:53.170693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:53.170654    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-47a8a4-88wff"]
Nov 03 11:27:53.224973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:53.224938    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:53.403254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:53.403200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:53.662878 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:53.662845    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:53.903601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:53.903530    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d6fab4-c76a-42b2-9aec-c7e46dcc5531" path="/var/lib/kubelet/pods/b2d6fab4-c76a-42b2-9aec-c7e46dcc5531/volumes"
Nov 03 11:27:54.154941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:54.154892    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:54.722915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:54.722872    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:54.955659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:54.945988    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:55.676391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:55.676358    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:56.364591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:56.364408    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:56.423513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:56.423486    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"]
Nov 03 11:27:56.614385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:56.614356    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:56.752476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:56.752443    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:57.030804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:57.030709    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:57.231075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:57.231039    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:58.005249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:58.005207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:58.675855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:58.675805    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:58.850592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:58.849977    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:59.520108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:59.519698    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:27:59.673088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:27:59.673048    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:00.333327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:00.333293    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:00.645833 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:00.645748    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:00.819613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:00.819544    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:01.004237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.004177    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:01.367852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.367777    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"]
Nov 03 11:28:01.368471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.368457    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7253e0f4-622a-47f7-afd3-c95f43a25420" containerName="revoke-keys"
Nov 03 11:28:01.368518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.368473    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7253e0f4-622a-47f7-afd3-c95f43a25420" containerName="revoke-keys"
Nov 03 11:28:01.368555 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.368528    5817 cpu_manager.go:414] "containerMap: removing container" podUID="aec2a5fd-6fc8-4f67-982f-c2cc457bc645" containerName="payload-processing"
Nov 03 11:28:01.368555 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.368533    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec2a5fd-6fc8-4f67-982f-c2cc457bc645" containerName="payload-processing"
Nov 03 11:28:01.368656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.368648    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec2a5fd-6fc8-4f67-982f-c2cc457bc645" containerName="payload-processing"
Nov 03 11:28:01.368697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.368657    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7253e0f4-622a-47f7-afd3-c95f43a25420" containerName="revoke-keys"
Nov 03 11:28:01.373683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.373664    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.380429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.380398    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"]
Nov 03 11:28:01.407004 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.406972    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.407190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.407099    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.407467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.407433    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7k97\" (UniqueName: \"kubernetes.io/projected/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-kube-api-access-f7k97\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.425762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.425724    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:28:01.426116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.426101    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:28:01.430999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.428082    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs" podUID="4fea08a2-6386-4412-a37c-4c75bd7c135e" containerName="payload-pre-processing" containerID="cri-o://b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570" gracePeriod=25
Nov 03 11:28:01.471778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.469197    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:01.509516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.509487    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7k97\" (UniqueName: \"kubernetes.io/projected/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-kube-api-access-f7k97\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.509741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.509555    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.509741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.509640    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.510344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.510318    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.512260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.512234    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.517632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.517607    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7k97\" (UniqueName: \"kubernetes.io/projected/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-kube-api-access-f7k97\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.702039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.702003    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:01.997698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:01.997669    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"]
Nov 03 11:28:01.998268 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:28:01.998240    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64bbb5d_d1c9_4497_8cc8_2af494ae142a.slice/crio-5b05183a0f35e3a643e9dcdbc37ddf5bf5bd3d0729abc19ca8ae158f5c1fe263 WatchSource:0}: Error finding container 5b05183a0f35e3a643e9dcdbc37ddf5bf5bd3d0729abc19ca8ae158f5c1fe263: Status 404 returned error can't find the container with id 5b05183a0f35e3a643e9dcdbc37ddf5bf5bd3d0729abc19ca8ae158f5c1fe263
Nov 03 11:28:02.278073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:02.277987    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57" event={"ID":"f64bbb5d-d1c9-4497-8cc8-2af494ae142a","Type":"ContainerStarted","Data":"5b05183a0f35e3a643e9dcdbc37ddf5bf5bd3d0729abc19ca8ae158f5c1fe263"}
Nov 03 11:28:02.344281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:02.344234    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:03.173656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:03.167391    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:03.294061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:03.294023    5817 generic.go:356] "Generic (PLEG): container finished" podID="f64bbb5d-d1c9-4497-8cc8-2af494ae142a" containerID="e226edd649524463aaf66a235f6f4e282358fa8427324b4a953a698799f8a371" exitCode=0
Nov 03 11:28:03.294258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:03.294180    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57" event={"ID":"f64bbb5d-d1c9-4497-8cc8-2af494ae142a","Type":"ContainerDied","Data":"e226edd649524463aaf66a235f6f4e282358fa8427324b4a953a698799f8a371"}
Nov 03 11:28:03.613476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:03.613446    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:28:04.076226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.070286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:04.466676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.466652    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:04.561101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.561059    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-kube-api-access-f7k97\" (UniqueName: \"kubernetes.io/projected/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-kube-api-access-f7k97\") pod \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") "
Nov 03 11:28:04.561341 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.561318    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-openshift-ca\") pod \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") "
Nov 03 11:28:04.561430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.561376    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-xks-ca\") pod \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\" (UID: \"f64bbb5d-d1c9-4497-8cc8-2af494ae142a\") "
Nov 03 11:28:04.565177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.565138    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-openshift-ca" pod "f64bbb5d-d1c9-4497-8cc8-2af494ae142a" (UID: "f64bbb5d-d1c9-4497-8cc8-2af494ae142a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:04.565317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.565233    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-kube-api-access-f7k97" pod "f64bbb5d-d1c9-4497-8cc8-2af494ae142a" (UID: "f64bbb5d-d1c9-4497-8cc8-2af494ae142a"). InnerVolumeSpecName "kube-api-access-f7k97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:04.569661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.569630    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-xks-ca" pod "f64bbb5d-d1c9-4497-8cc8-2af494ae142a" (UID: "f64bbb5d-d1c9-4497-8cc8-2af494ae142a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:28:04.664654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.663914    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:04.664654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.663958    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:04.664654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.663982    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7k97\" (UniqueName: \"kubernetes.io/projected/f64bbb5d-d1c9-4497-8cc8-2af494ae142a-kube-api-access-f7k97\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:04.922081 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:04.921989    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:05.313905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:05.313876    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"
Nov 03 11:28:05.313905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:05.313891    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57" event={"ID":"f64bbb5d-d1c9-4497-8cc8-2af494ae142a","Type":"ContainerDied","Data":"5b05183a0f35e3a643e9dcdbc37ddf5bf5bd3d0729abc19ca8ae158f5c1fe263"}
Nov 03 11:28:05.314209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:05.313968    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b05183a0f35e3a643e9dcdbc37ddf5bf5bd3d0729abc19ca8ae158f5c1fe263"
Nov 03 11:28:05.694632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:05.694541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:06.420610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:06.419597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:07.164470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:07.164434    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:08.057606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:08.057545    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:08.651419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:08.651374    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:08.778256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:08.778219    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"]
Nov 03 11:28:08.783425 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:08.783392    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cbeebd-2kfj8"]
Nov 03 11:28:08.792207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:08.792162    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:08.797625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:08.797597    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs" podUID="4fea08a2-6386-4412-a37c-4c75bd7c135e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:08.939514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:08.939470    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:09.311865 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:09.311775    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:09.464040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:09.464003    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:09.902904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:09.902873    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7253e0f4-622a-47f7-afd3-c95f43a25420" path="/var/lib/kubelet/pods/7253e0f4-622a-47f7-afd3-c95f43a25420/volumes"
Nov 03 11:28:10.118017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:10.117980    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:10.879183 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:10.877465    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:11.535225 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:11.535195    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"]
Nov 03 11:28:11.556440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:11.556408    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"]
Nov 03 11:28:11.594518 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:11.594466    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:11.736496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:11.736464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:12.416262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:12.416224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:13.112516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:13.112480    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:13.776253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:13.776214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:14.600434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:14.600393    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:15.278599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:15.276871    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:15.732476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:15.732442    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:15.881370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:15.881335    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:15.998196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:15.998105    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"]
Nov 03 11:28:15.998388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:15.998356    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy" containerID="cri-o://f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715" gracePeriod=30
Nov 03 11:28:16.028985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.028944    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:16.193750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.193704    5817 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:28:16.194057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.194029    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:28:16.245100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.245034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:16.537957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.537673    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn" podUID="dda1d70a-1612-4004-93bd-7ce1a4f3a28e" containerName="payload-processing" containerID="cri-o://06c0d32b0e7d8b3b941d2ab37ee2006686877405b47a99316b561d1bfb321120" gracePeriod=25
Nov 03 11:28:16.542595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.538267    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"]
Nov 03 11:28:16.542595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.540309    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f64bbb5d-d1c9-4497-8cc8-2af494ae142a" containerName="revoke-keys"
Nov 03 11:28:16.542595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.540331    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f64bbb5d-d1c9-4497-8cc8-2af494ae142a" containerName="revoke-keys"
Nov 03 11:28:16.542595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.540815    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f64bbb5d-d1c9-4497-8cc8-2af494ae142a" containerName="revoke-keys"
Nov 03 11:28:16.551961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.551888    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.552343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.552310    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"]
Nov 03 11:28:16.557883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.557848    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs" podUID="f52cc2fc-370d-4a93-a725-225d36d0ec5d" containerName="payload-pre-processing" containerID="cri-o://4369926e61ac697fe7c3cbb71df5829b721416a0bd4f01d192dc33e72ffd1b4f" gracePeriod=25
Nov 03 11:28:16.573036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.573010    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92x5t\" (UniqueName: \"kubernetes.io/projected/77435310-5f08-466a-a9ab-9d92e9bafb7f-kube-api-access-92x5t\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.573170 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.573073    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.573373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.573312    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.618646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.618623    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:28:16.618767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.618681    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:28:16.675072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.675041    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.675227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.675213    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92x5t\" (UniqueName: \"kubernetes.io/projected/77435310-5f08-466a-a9ab-9d92e9bafb7f-kube-api-access-92x5t\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.675298 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.675283    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.675703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.675682    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.677834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.677815    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.683197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.683177    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92x5t\" (UniqueName: \"kubernetes.io/projected/77435310-5f08-466a-a9ab-9d92e9bafb7f-kube-api-access-92x5t\") pod \"maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:16.886670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.880158    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:16.886670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:16.882802    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:17.193754 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:28:17.193722    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77435310_5f08_466a_a9ab_9d92e9bafb7f.slice/crio-91c6881b68f68cc10b6b78301d38977dce32b71012c839505d6d14608274ff5c WatchSource:0}: Error finding container 91c6881b68f68cc10b6b78301d38977dce32b71012c839505d6d14608274ff5c: Status 404 returned error can't find the container with id 91c6881b68f68cc10b6b78301d38977dce32b71012c839505d6d14608274ff5c
Nov 03 11:28:17.194287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:17.194235    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"]
Nov 03 11:28:17.472246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:17.472150    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp" event={"ID":"77435310-5f08-466a-a9ab-9d92e9bafb7f","Type":"ContainerStarted","Data":"91c6881b68f68cc10b6b78301d38977dce32b71012c839505d6d14608274ff5c"}
Nov 03 11:28:17.568609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:17.566418    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:18.293831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:18.290208    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:18.481936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:18.481906    5817 generic.go:356] "Generic (PLEG): container finished" podID="77435310-5f08-466a-a9ab-9d92e9bafb7f" containerID="cbeab8ab754b60c107f703258bc74731ad252f8696163c714475052b2cce3b1b" exitCode=0
Nov 03 11:28:18.482098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:18.481996    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp" event={"ID":"77435310-5f08-466a-a9ab-9d92e9bafb7f","Type":"ContainerDied","Data":"cbeab8ab754b60c107f703258bc74731ad252f8696163c714475052b2cce3b1b"}
Nov 03 11:28:18.796821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:18.796783    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs" podUID="4fea08a2-6386-4412-a37c-4c75bd7c135e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:19.109535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.108741    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:19.665963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.665931    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:19.737271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.737244    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-openshift-ca\") pod \"77435310-5f08-466a-a9ab-9d92e9bafb7f\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") "
Nov 03 11:28:19.737485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.737451    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77435310-5f08-466a-a9ab-9d92e9bafb7f-kube-api-access-92x5t\" (UniqueName: \"kubernetes.io/projected/77435310-5f08-466a-a9ab-9d92e9bafb7f-kube-api-access-92x5t\") pod \"77435310-5f08-466a-a9ab-9d92e9bafb7f\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") "
Nov 03 11:28:19.737485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.737473    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-xks-ca\") pod \"77435310-5f08-466a-a9ab-9d92e9bafb7f\" (UID: \"77435310-5f08-466a-a9ab-9d92e9bafb7f\") "
Nov 03 11:28:19.737997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.737972    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-openshift-ca" pod "77435310-5f08-466a-a9ab-9d92e9bafb7f" (UID: "77435310-5f08-466a-a9ab-9d92e9bafb7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:19.741000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.740943    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-xks-ca" pod "77435310-5f08-466a-a9ab-9d92e9bafb7f" (UID: "77435310-5f08-466a-a9ab-9d92e9bafb7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:28:19.741181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.741160    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77435310-5f08-466a-a9ab-9d92e9bafb7f-kube-api-access-92x5t" pod "77435310-5f08-466a-a9ab-9d92e9bafb7f" (UID: "77435310-5f08-466a-a9ab-9d92e9bafb7f"). InnerVolumeSpecName "kube-api-access-92x5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:19.839205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.839175    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:19.839205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.839203    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92x5t\" (UniqueName: \"kubernetes.io/projected/77435310-5f08-466a-a9ab-9d92e9bafb7f-kube-api-access-92x5t\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:19.839421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.839213    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77435310-5f08-466a-a9ab-9d92e9bafb7f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:19.892613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:19.879449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:20.501748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:20.501665    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp" event={"ID":"77435310-5f08-466a-a9ab-9d92e9bafb7f","Type":"ContainerDied","Data":"91c6881b68f68cc10b6b78301d38977dce32b71012c839505d6d14608274ff5c"}
Nov 03 11:28:20.501748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:20.501685    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"
Nov 03 11:28:20.501748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:20.501721    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c6881b68f68cc10b6b78301d38977dce32b71012c839505d6d14608274ff5c"
Nov 03 11:28:20.591087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:20.579831    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:21.116984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.116956    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"]
Nov 03 11:28:21.117757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.117726    5817 cpu_manager.go:414] "containerMap: removing container" podUID="77435310-5f08-466a-a9ab-9d92e9bafb7f" containerName="revoke-keys"
Nov 03 11:28:21.117757 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.117749    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77435310-5f08-466a-a9ab-9d92e9bafb7f" containerName="revoke-keys"
Nov 03 11:28:21.117936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.117885    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="77435310-5f08-466a-a9ab-9d92e9bafb7f" containerName="revoke-keys"
Nov 03 11:28:21.150424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.150376    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"]
Nov 03 11:28:21.150656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.150501    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.160795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.160767    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:28:21.161147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.161129    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:28:21.279552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.279051    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.279552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.279168    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.279831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.279625    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t46p\" (UniqueName: \"kubernetes.io/projected/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-kube-api-access-4t46p\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.279831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.279652    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:21.361633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.361596    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:28:21.380822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.380798    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.380991 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.380977    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t46p\" (UniqueName: \"kubernetes.io/projected/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-kube-api-access-4t46p\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.381056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.381045    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.381654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.381600    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.383219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.383198    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.400180 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.400155    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t46p\" (UniqueName: \"kubernetes.io/projected/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-kube-api-access-4t46p\") pod \"maas-api-revoke-sub-9023c992-vkmw8\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.469847 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.469819    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:21.482062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482034    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-envoy\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482118    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-data\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482238    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-kube-api-access-q2pgs\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-kube-api-access-q2pgs\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482272    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-token\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-token\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482329    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-podinfo\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482359    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-socket\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482385    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-data" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:21.482537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482415    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istiod-ca-cert\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482458    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-credential-socket\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482553    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-certs\") pod \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\" (UID: \"f906dbec-7c86-4e5a-a394-ff8a83fb03c1\") "
Nov 03 11:28:21.482914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.482888    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istiod-ca-cert" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:21.483066 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.483021    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-socket" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:21.483267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.483232    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-certs" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:21.483267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.483248    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-credential-socket" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:21.483722 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.483666    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.483835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.483729    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.483835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.483746    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.483835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.483760    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.483835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.483775    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.484983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.484951    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-envoy" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:21.485111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.485087    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-podinfo" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:28:21.485184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.485152    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-kube-api-access-q2pgs" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "kube-api-access-q2pgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:21.485235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.485181    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-token" pod "f906dbec-7c86-4e5a-a394-ff8a83fb03c1" (UID: "f906dbec-7c86-4e5a-a394-ff8a83fb03c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:21.513949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.512690    5817 generic.go:356] "Generic (PLEG): container finished" podID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerID="f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715" exitCode=0
Nov 03 11:28:21.513949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.512932    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" event={"ID":"f906dbec-7c86-4e5a-a394-ff8a83fb03c1","Type":"ContainerDied","Data":"f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715"}
Nov 03 11:28:21.513949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.512969    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2" event={"ID":"f906dbec-7c86-4e5a-a394-ff8a83fb03c1","Type":"ContainerDied","Data":"f2148c4982a5a43ff11a8132d4d7e9de4e8319a7acf3d29aa7644bfb3fcf3182"}
Nov 03 11:28:21.513949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.512992    5817 scope.go:122] "RemoveContainer" containerID="f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715"
Nov 03 11:28:21.513949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.513142    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"
Nov 03 11:28:21.545921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.545772    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"]
Nov 03 11:28:21.556825 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.553381    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-v7rj2"]
Nov 03 11:28:21.559531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.559503    5817 scope.go:122] "RemoveContainer" containerID="10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668"
Nov 03 11:28:21.578467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.578427    5817 scope.go:122] "RemoveContainer" containerID="f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715"
Nov 03 11:28:21.578949 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:28:21.578919    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715\": container with ID starting with f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715 not found: ID does not exist" containerID="f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715"
Nov 03 11:28:21.579083 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.578959    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715"} err="failed to get container status \"f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715\": rpc error: code = NotFound desc = could not find container \"f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715\": container with ID starting with f8e5ac92247cd64b877c3e3a38d9d12c44d60424010ae739783e7b71de73f715 not found: ID does not exist"
Nov 03 11:28:21.579083 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.578983    5817 scope.go:122] "RemoveContainer" containerID="10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668"
Nov 03 11:28:21.579360 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:28:21.579286    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668\": container with ID starting with 10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668 not found: ID does not exist" containerID="10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668"
Nov 03 11:28:21.579360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.579319    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668"} err="failed to get container status \"10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668\": rpc error: code = NotFound desc = could not find container \"10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668\": container with ID starting with 10719b63ff02010987bbd153dfc723027eda1348f10a14a82dccba856099b668 not found: ID does not exist"
Nov 03 11:28:21.585018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.584996    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.585106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.585019    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2pgs\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-kube-api-access-q2pgs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.585106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.585031    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.585106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.585044    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f906dbec-7c86-4e5a-a394-ff8a83fb03c1-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.733469 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.733439    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:28:21.787565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.787479    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fea08a2-6386-4412-a37c-4c75bd7c135e-kube-api-access-7w75m\" (UniqueName: \"kubernetes.io/projected/4fea08a2-6386-4412-a37c-4c75bd7c135e-kube-api-access-7w75m\") pod \"4fea08a2-6386-4412-a37c-4c75bd7c135e\" (UID: \"4fea08a2-6386-4412-a37c-4c75bd7c135e\") "
Nov 03 11:28:21.787746 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.787625    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fea08a2-6386-4412-a37c-4c75bd7c135e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fea08a2-6386-4412-a37c-4c75bd7c135e-plugins-config-volume\") pod \"4fea08a2-6386-4412-a37c-4c75bd7c135e\" (UID: \"4fea08a2-6386-4412-a37c-4c75bd7c135e\") "
Nov 03 11:28:21.790188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.789339    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fea08a2-6386-4412-a37c-4c75bd7c135e-plugins-config-volume" pod "4fea08a2-6386-4412-a37c-4c75bd7c135e" (UID: "4fea08a2-6386-4412-a37c-4c75bd7c135e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:21.793198 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.793169    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fea08a2-6386-4412-a37c-4c75bd7c135e-kube-api-access-7w75m" pod "4fea08a2-6386-4412-a37c-4c75bd7c135e" (UID: "4fea08a2-6386-4412-a37c-4c75bd7c135e"). InnerVolumeSpecName "kube-api-access-7w75m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:21.806393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.806363    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"]
Nov 03 11:28:21.806789 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:28:21.806762    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a4e895_c827_4437_9ae9_c1ba8d0bc55e.slice/crio-9eac119db96304293bb059724db1cf37d5fae58534c7374d2d6a1939ef4bb811 WatchSource:0}: Error finding container 9eac119db96304293bb059724db1cf37d5fae58534c7374d2d6a1939ef4bb811: Status 404 returned error can't find the container with id 9eac119db96304293bb059724db1cf37d5fae58534c7374d2d6a1939ef4bb811
Nov 03 11:28:21.890021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.889987    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w75m\" (UniqueName: \"kubernetes.io/projected/4fea08a2-6386-4412-a37c-4c75bd7c135e-kube-api-access-7w75m\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.890021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.890019    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fea08a2-6386-4412-a37c-4c75bd7c135e-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.905038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.904992    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" path="/var/lib/kubelet/pods/f906dbec-7c86-4e5a-a394-ff8a83fb03c1/volumes"
Nov 03 11:28:21.934159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:21.934121    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:22.145596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.145030    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:22.299990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.297698    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"]
Nov 03 11:28:22.299990 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.298409    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:22.309885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.309852    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e11de4-npv57"]
Nov 03 11:28:22.434194 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:28:22.434160    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:28:22.440078 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.440047    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:22.524687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.524654    5817 generic.go:356] "Generic (PLEG): container finished" podID="4fea08a2-6386-4412-a37c-4c75bd7c135e" containerID="b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570" exitCode=0
Nov 03 11:28:22.524857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.524695    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs" event={"ID":"4fea08a2-6386-4412-a37c-4c75bd7c135e","Type":"ContainerDied","Data":"b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570"}
Nov 03 11:28:22.524857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.524707    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"
Nov 03 11:28:22.524857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.524749    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs" event={"ID":"4fea08a2-6386-4412-a37c-4c75bd7c135e","Type":"ContainerDied","Data":"df1a5362f8f8830dfd764f045cbda542e9f2f8e4cc64bf9b364076f9f754c5c8"}
Nov 03 11:28:22.524857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.524778    5817 scope.go:122] "RemoveContainer" containerID="b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570"
Nov 03 11:28:22.529246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.529222    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8" event={"ID":"77a4e895-c827-4437-9ae9-c1ba8d0bc55e","Type":"ContainerStarted","Data":"418542a31f529ec867401e84aef632fff628b302195fb6788c427429f1a79d81"}
Nov 03 11:28:22.529382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.529266    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8" event={"ID":"77a4e895-c827-4437-9ae9-c1ba8d0bc55e","Type":"ContainerStarted","Data":"9eac119db96304293bb059724db1cf37d5fae58534c7374d2d6a1939ef4bb811"}
Nov 03 11:28:22.537587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.537550    5817 scope.go:122] "RemoveContainer" containerID="b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570"
Nov 03 11:28:22.537893 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:28:22.537863    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570\": container with ID starting with b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570 not found: ID does not exist" containerID="b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570"
Nov 03 11:28:22.538011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.537895    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570"} err="failed to get container status \"b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570\": rpc error: code = NotFound desc = could not find container \"b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570\": container with ID starting with b8e9b4e17b81ece4cd8fb6e133da12401363c80d2e94cb5f152ceff02ea32570 not found: ID does not exist"
Nov 03 11:28:22.544753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.544734    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"]
Nov 03 11:28:22.549788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.549766    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cbeebd-85b755db8c-vrdvs"]
Nov 03 11:28:22.580516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:22.580444    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8" podStartSLOduration=1.580431508 podStartE2EDuration="1.580431508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:28:22.578677819 +0000 UTC m=+3532.784259871" watchObservedRunningTime="2026-11-03 11:28:22.580431508 +0000 UTC m=+3532.786013559"
Nov 03 11:28:23.367460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:23.367422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:23.540587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:23.540541    5817 generic.go:356] "Generic (PLEG): container finished" podID="77a4e895-c827-4437-9ae9-c1ba8d0bc55e" containerID="418542a31f529ec867401e84aef632fff628b302195fb6788c427429f1a79d81" exitCode=0
Nov 03 11:28:23.540749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:23.540625    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8" event={"ID":"77a4e895-c827-4437-9ae9-c1ba8d0bc55e","Type":"ContainerDied","Data":"418542a31f529ec867401e84aef632fff628b302195fb6788c427429f1a79d81"}
Nov 03 11:28:23.713464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:23.713422    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn" podUID="dda1d70a-1612-4004-93bd-7ce1a4f3a28e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:23.713915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:23.713885    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs" podUID="f52cc2fc-370d-4a93-a725-225d36d0ec5d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:23.903954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:23.903920    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fea08a2-6386-4412-a37c-4c75bd7c135e" path="/var/lib/kubelet/pods/4fea08a2-6386-4412-a37c-4c75bd7c135e/volumes"
Nov 03 11:28:23.904463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:23.904441    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f64bbb5d-d1c9-4497-8cc8-2af494ae142a" path="/var/lib/kubelet/pods/f64bbb5d-d1c9-4497-8cc8-2af494ae142a/volumes"
Nov 03 11:28:24.178651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.176158    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:24.731057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.731032    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:24.843364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.843281    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-kube-api-access-4t46p\" (UniqueName: \"kubernetes.io/projected/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-kube-api-access-4t46p\") pod \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") "
Nov 03 11:28:24.843364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.843320    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-openshift-ca\") pod \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") "
Nov 03 11:28:24.843627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.843508    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-xks-ca\") pod \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\" (UID: \"77a4e895-c827-4437-9ae9-c1ba8d0bc55e\") "
Nov 03 11:28:24.843843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.843817    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-openshift-ca" pod "77a4e895-c827-4437-9ae9-c1ba8d0bc55e" (UID: "77a4e895-c827-4437-9ae9-c1ba8d0bc55e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:24.844132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.844113    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:24.845852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.845822    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-xks-ca" pod "77a4e895-c827-4437-9ae9-c1ba8d0bc55e" (UID: "77a4e895-c827-4437-9ae9-c1ba8d0bc55e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:28:24.846275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.846248    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-kube-api-access-4t46p" pod "77a4e895-c827-4437-9ae9-c1ba8d0bc55e" (UID: "77a4e895-c827-4437-9ae9-c1ba8d0bc55e"). InnerVolumeSpecName "kube-api-access-4t46p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:24.945270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.945237    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:24.945270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:24.945269    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t46p\" (UniqueName: \"kubernetes.io/projected/77a4e895-c827-4437-9ae9-c1ba8d0bc55e-kube-api-access-4t46p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:25.192872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:25.192836    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:25.562401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:25.562314    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"
Nov 03 11:28:25.562401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:25.562336    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8" event={"ID":"77a4e895-c827-4437-9ae9-c1ba8d0bc55e","Type":"ContainerDied","Data":"9eac119db96304293bb059724db1cf37d5fae58534c7374d2d6a1939ef4bb811"}
Nov 03 11:28:25.562654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:25.562408    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eac119db96304293bb059724db1cf37d5fae58534c7374d2d6a1939ef4bb811"
Nov 03 11:28:26.199177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:26.199066    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:26.940222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:26.940187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:27.121597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:27.119546    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:27.897215 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:27.897179    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:28.791043 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:28.787731    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:29.538594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:29.535526    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:30.304593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:30.301164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:30.996676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:30.996643    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:31.381565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.381489    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"]
Nov 03 11:28:31.382150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382136    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy"
Nov 03 11:28:31.382197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382152    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy"
Nov 03 11:28:31.382197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382174    5817 cpu_manager.go:414] "containerMap: removing container" podUID="77a4e895-c827-4437-9ae9-c1ba8d0bc55e" containerName="revoke-keys"
Nov 03 11:28:31.382197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382180    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a4e895-c827-4437-9ae9-c1ba8d0bc55e" containerName="revoke-keys"
Nov 03 11:28:31.382297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382200    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy"
Nov 03 11:28:31.382297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382205    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy"
Nov 03 11:28:31.382297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382230    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4fea08a2-6386-4412-a37c-4c75bd7c135e" containerName="payload-pre-processing"
Nov 03 11:28:31.382297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382235    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fea08a2-6386-4412-a37c-4c75bd7c135e" containerName="payload-pre-processing"
Nov 03 11:28:31.382433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382321    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy"
Nov 03 11:28:31.382433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382334    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fea08a2-6386-4412-a37c-4c75bd7c135e" containerName="payload-pre-processing"
Nov 03 11:28:31.382433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.382344    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a4e895-c827-4437-9ae9-c1ba8d0bc55e" containerName="revoke-keys"
Nov 03 11:28:31.386212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.386196    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.390795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.390742    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:28:31.391625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.391564    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:28:31.394605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.394528    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"]
Nov 03 11:28:31.563173 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.563141    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.563349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.563200    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.563349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.563314    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzfp\" (UniqueName: \"kubernetes.io/projected/49c95484-2e1e-4567-b5a6-73da33e4ca1f-kube-api-access-dxzfp\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.664353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.664283    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.664353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.664343    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzfp\" (UniqueName: \"kubernetes.io/projected/49c95484-2e1e-4567-b5a6-73da33e4ca1f-kube-api-access-dxzfp\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.664544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.664480    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.665046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.665028    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.667000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.666978    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.673302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.673278    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzfp\" (UniqueName: \"kubernetes.io/projected/49c95484-2e1e-4567-b5a6-73da33e4ca1f-kube-api-access-dxzfp\") pod \"maas-api-revoke-sub-54aad279-fpm7j\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.707493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.707469    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:31.774406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:31.774377    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:32.050135 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:28:32.050107    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c95484_2e1e_4567_b5a6_73da33e4ca1f.slice/crio-71bdc009d23981b02b14a677393afc6ebe5996dfb97d3fdef160ffbbda6b162d WatchSource:0}: Error finding container 71bdc009d23981b02b14a677393afc6ebe5996dfb97d3fdef160ffbbda6b162d: Status 404 returned error can't find the container with id 71bdc009d23981b02b14a677393afc6ebe5996dfb97d3fdef160ffbbda6b162d
Nov 03 11:28:32.050606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:32.050180    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"]
Nov 03 11:28:32.521107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:32.521067    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:32.641096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:32.641023    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j" event={"ID":"49c95484-2e1e-4567-b5a6-73da33e4ca1f","Type":"ContainerStarted","Data":"a34f8e16e5975dd0ac5ddddf86543e4ddf81c44fe80b019e96a92869dbd259f9"}
Nov 03 11:28:32.641096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:32.641079    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j" event={"ID":"49c95484-2e1e-4567-b5a6-73da33e4ca1f","Type":"ContainerStarted","Data":"71bdc009d23981b02b14a677393afc6ebe5996dfb97d3fdef160ffbbda6b162d"}
Nov 03 11:28:32.660692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:32.660649    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j" podStartSLOduration=1.6606375469999999 podStartE2EDuration="1.660637547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:28:32.658117121 +0000 UTC m=+3542.863699172" watchObservedRunningTime="2026-11-03 11:28:32.660637547 +0000 UTC m=+3542.866219642"
Nov 03 11:28:32.890688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:32.890648    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:33.175546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:33.164189    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:33.651397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:33.651363    5817 generic.go:356] "Generic (PLEG): container finished" podID="49c95484-2e1e-4567-b5a6-73da33e4ca1f" containerID="a34f8e16e5975dd0ac5ddddf86543e4ddf81c44fe80b019e96a92869dbd259f9" exitCode=0
Nov 03 11:28:33.651616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:33.651445    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j" event={"ID":"49c95484-2e1e-4567-b5a6-73da33e4ca1f","Type":"ContainerDied","Data":"a34f8e16e5975dd0ac5ddddf86543e4ddf81c44fe80b019e96a92869dbd259f9"}
Nov 03 11:28:33.713455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:33.713418    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn" podUID="dda1d70a-1612-4004-93bd-7ce1a4f3a28e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:33.714353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:33.714323    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs" podUID="f52cc2fc-370d-4a93-a725-225d36d0ec5d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:33.892834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:33.892796    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:34.681045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:34.680997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:34.862403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:34.862380    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:34.918589 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:34.918532    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-xks-ca\") pod \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") "
Nov 03 11:28:34.918778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:34.918666    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-openshift-ca\") pod \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") "
Nov 03 11:28:34.919950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:34.919923    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-openshift-ca" pod "49c95484-2e1e-4567-b5a6-73da33e4ca1f" (UID: "49c95484-2e1e-4567-b5a6-73da33e4ca1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:34.925053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:34.925024    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-xks-ca" pod "49c95484-2e1e-4567-b5a6-73da33e4ca1f" (UID: "49c95484-2e1e-4567-b5a6-73da33e4ca1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:28:35.019900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.019866    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c95484-2e1e-4567-b5a6-73da33e4ca1f-kube-api-access-dxzfp\" (UniqueName: \"kubernetes.io/projected/49c95484-2e1e-4567-b5a6-73da33e4ca1f-kube-api-access-dxzfp\") pod \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\" (UID: \"49c95484-2e1e-4567-b5a6-73da33e4ca1f\") "
Nov 03 11:28:35.020649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.020230    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:35.020649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.020254    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c95484-2e1e-4567-b5a6-73da33e4ca1f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:35.022519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.022491    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c95484-2e1e-4567-b5a6-73da33e4ca1f-kube-api-access-dxzfp" pod "49c95484-2e1e-4567-b5a6-73da33e4ca1f" (UID: "49c95484-2e1e-4567-b5a6-73da33e4ca1f"). InnerVolumeSpecName "kube-api-access-dxzfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:35.121297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.121264    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxzfp\" (UniqueName: \"kubernetes.io/projected/49c95484-2e1e-4567-b5a6-73da33e4ca1f-kube-api-access-dxzfp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:35.558551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.558516    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:35.693499 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.693465    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"
Nov 03 11:28:35.693919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.693499    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j" event={"ID":"49c95484-2e1e-4567-b5a6-73da33e4ca1f","Type":"ContainerDied","Data":"71bdc009d23981b02b14a677393afc6ebe5996dfb97d3fdef160ffbbda6b162d"}
Nov 03 11:28:35.693919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.693622    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71bdc009d23981b02b14a677393afc6ebe5996dfb97d3fdef160ffbbda6b162d"
Nov 03 11:28:35.702822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.702684    5817 scope.go:122] "RemoveContainer" containerID="e4ab3554cec4f2678797142b16e3873af8a60a05df2dd7ada8729796ac859ab8"
Nov 03 11:28:35.715880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.715860    5817 scope.go:122] "RemoveContainer" containerID="b780b71fa03313594a614fd9ee20374635d8d1ff034f293147181f0518959096"
Nov 03 11:28:35.726235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.726145    5817 scope.go:122] "RemoveContainer" containerID="cd79a6b03f01c7990ff94b71d73f17f5907ec563b5d8f4b2519646d9b416450d"
Nov 03 11:28:35.737642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:35.737618    5817 scope.go:122] "RemoveContainer" containerID="7f9b56d597bc4789b9cc8d3aa60ad919244f22684025be6cc179ef1c2dcba3a6"
Nov 03 11:28:36.165601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.163218    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:36.710594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.709770    5817 generic.go:356] "Generic (PLEG): container finished" podID="dda1d70a-1612-4004-93bd-7ce1a4f3a28e" containerID="06c0d32b0e7d8b3b941d2ab37ee2006686877405b47a99316b561d1bfb321120" exitCode=0
Nov 03 11:28:36.710594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.709907    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn" event={"ID":"dda1d70a-1612-4004-93bd-7ce1a4f3a28e","Type":"ContainerDied","Data":"06c0d32b0e7d8b3b941d2ab37ee2006686877405b47a99316b561d1bfb321120"}
Nov 03 11:28:36.725592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.716624    5817 generic.go:356] "Generic (PLEG): container finished" podID="f52cc2fc-370d-4a93-a725-225d36d0ec5d" containerID="4369926e61ac697fe7c3cbb71df5829b721416a0bd4f01d192dc33e72ffd1b4f" exitCode=0
Nov 03 11:28:36.725592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.716759    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs" event={"ID":"f52cc2fc-370d-4a93-a725-225d36d0ec5d","Type":"ContainerDied","Data":"4369926e61ac697fe7c3cbb71df5829b721416a0bd4f01d192dc33e72ffd1b4f"}
Nov 03 11:28:36.831802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.826406    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:36.901994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.901965    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 11:28:36.947613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.947560    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 11:28:36.954759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.954721    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52cc2fc-370d-4a93-a725-225d36d0ec5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f52cc2fc-370d-4a93-a725-225d36d0ec5d-plugins-config-volume\") pod \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\" (UID: \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\") "
Nov 03 11:28:36.956363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.954951    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52cc2fc-370d-4a93-a725-225d36d0ec5d-kube-api-access-p76n5\" (UniqueName: \"kubernetes.io/projected/f52cc2fc-370d-4a93-a725-225d36d0ec5d-kube-api-access-p76n5\") pod \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\" (UID: \"f52cc2fc-370d-4a93-a725-225d36d0ec5d\") "
Nov 03 11:28:36.956363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.955088    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52cc2fc-370d-4a93-a725-225d36d0ec5d-plugins-config-volume" pod "f52cc2fc-370d-4a93-a725-225d36d0ec5d" (UID: "f52cc2fc-370d-4a93-a725-225d36d0ec5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:36.956363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.955484    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f52cc2fc-370d-4a93-a725-225d36d0ec5d-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:36.957410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:36.957390    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52cc2fc-370d-4a93-a725-225d36d0ec5d-kube-api-access-p76n5" pod "f52cc2fc-370d-4a93-a725-225d36d0ec5d" (UID: "f52cc2fc-370d-4a93-a725-225d36d0ec5d"). InnerVolumeSpecName "kube-api-access-p76n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:37.056256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.056223    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-kube-api-access-x7t2s\" (UniqueName: \"kubernetes.io/projected/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-kube-api-access-x7t2s\") pod \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\" (UID: \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\") "
Nov 03 11:28:37.056429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.056375    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-plugins-config-volume\") pod \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\" (UID: \"dda1d70a-1612-4004-93bd-7ce1a4f3a28e\") "
Nov 03 11:28:37.056727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.056701    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76n5\" (UniqueName: \"kubernetes.io/projected/f52cc2fc-370d-4a93-a725-225d36d0ec5d-kube-api-access-p76n5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:37.056853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.056761    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-plugins-config-volume" pod "dda1d70a-1612-4004-93bd-7ce1a4f3a28e" (UID: "dda1d70a-1612-4004-93bd-7ce1a4f3a28e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:37.058451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.058433    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-kube-api-access-x7t2s" pod "dda1d70a-1612-4004-93bd-7ce1a4f3a28e" (UID: "dda1d70a-1612-4004-93bd-7ce1a4f3a28e"). InnerVolumeSpecName "kube-api-access-x7t2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:37.157907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.157832    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:37.157907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.157859    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7t2s\" (UniqueName: \"kubernetes.io/projected/dda1d70a-1612-4004-93bd-7ce1a4f3a28e-kube-api-access-x7t2s\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:37.511412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.511358    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:37.732635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.731102    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"
Nov 03 11:28:37.732635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.731101    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn" event={"ID":"dda1d70a-1612-4004-93bd-7ce1a4f3a28e","Type":"ContainerDied","Data":"cc0eb0684759083f6773863c76105ddd5a2ae396ef2fe52890bb512236fc354b"}
Nov 03 11:28:37.732635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.731265    5817 scope.go:122] "RemoveContainer" containerID="06c0d32b0e7d8b3b941d2ab37ee2006686877405b47a99316b561d1bfb321120"
Nov 03 11:28:37.733404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.733274    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs" event={"ID":"f52cc2fc-370d-4a93-a725-225d36d0ec5d","Type":"ContainerDied","Data":"2ab9c04ca1f0c2bccdcf2ad5086f7f2dd23bc893534ac44bfd7725b2a813ad75"}
Nov 03 11:28:37.733404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.733363    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"
Nov 03 11:28:37.745420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.745401    5817 scope.go:122] "RemoveContainer" containerID="4369926e61ac697fe7c3cbb71df5829b721416a0bd4f01d192dc33e72ffd1b4f"
Nov 03 11:28:37.761726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.761670    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"]
Nov 03 11:28:37.765964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.765944    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e11de4-549d7f476c-jvlfn"]
Nov 03 11:28:37.775204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.775176    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"]
Nov 03 11:28:37.778718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.778698    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e11de4-59444986f9-5f2gs"]
Nov 03 11:28:37.904138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.904097    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda1d70a-1612-4004-93bd-7ce1a4f3a28e" path="/var/lib/kubelet/pods/dda1d70a-1612-4004-93bd-7ce1a4f3a28e/volumes"
Nov 03 11:28:37.904717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:37.904697    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52cc2fc-370d-4a93-a725-225d36d0ec5d" path="/var/lib/kubelet/pods/f52cc2fc-370d-4a93-a725-225d36d0ec5d/volumes"
Nov 03 11:28:38.191594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:38.184103    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:38.812332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:38.812296    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:39.116774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:39.116676    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:39.251651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:39.251614    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:39.576655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:39.572997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:39.731667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:39.731633    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:39.999891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:39.999846    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:40.111074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.111044    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"]
Nov 03 11:28:40.112263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.112236    5817 cpu_manager.go:414] "containerMap: removing container" podUID="49c95484-2e1e-4567-b5a6-73da33e4ca1f" containerName="revoke-keys"
Nov 03 11:28:40.112428 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.112416    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c95484-2e1e-4567-b5a6-73da33e4ca1f" containerName="revoke-keys"
Nov 03 11:28:40.112521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.112511    5817 cpu_manager.go:414] "containerMap: removing container" podUID="dda1d70a-1612-4004-93bd-7ce1a4f3a28e" containerName="payload-processing"
Nov 03 11:28:40.112612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.112602    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda1d70a-1612-4004-93bd-7ce1a4f3a28e" containerName="payload-processing"
Nov 03 11:28:40.112763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.112754    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f52cc2fc-370d-4a93-a725-225d36d0ec5d" containerName="payload-pre-processing"
Nov 03 11:28:40.112835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.112826    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52cc2fc-370d-4a93-a725-225d36d0ec5d" containerName="payload-pre-processing"
Nov 03 11:28:40.113069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.113059    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda1d70a-1612-4004-93bd-7ce1a4f3a28e" containerName="payload-processing"
Nov 03 11:28:40.113158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.113150    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f906dbec-7c86-4e5a-a394-ff8a83fb03c1" containerName="istio-proxy"
Nov 03 11:28:40.113230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.113222    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52cc2fc-370d-4a93-a725-225d36d0ec5d" containerName="payload-pre-processing"
Nov 03 11:28:40.113304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.113297    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c95484-2e1e-4567-b5a6-73da33e4ca1f" containerName="revoke-keys"
Nov 03 11:28:40.119095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.119063    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.134222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.134196    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"]
Nov 03 11:28:40.157493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.157458    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:40.165522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.165495    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:28:40.165918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.165898    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:28:40.237767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.237719    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.237949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.237828    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sr6s\" (UniqueName: \"kubernetes.io/projected/4411f11b-f8a9-4a8e-824d-6319472a1963-kube-api-access-9sr6s\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.237949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.237883    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.339610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.339491    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.339796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.339639    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sr6s\" (UniqueName: \"kubernetes.io/projected/4411f11b-f8a9-4a8e-824d-6319472a1963-kube-api-access-9sr6s\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.339796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.339692    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.340524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.340499    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.342204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.342177    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.355638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.355614    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sr6s\" (UniqueName: \"kubernetes.io/projected/4411f11b-f8a9-4a8e-824d-6319472a1963-kube-api-access-9sr6s\") pod \"maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.501787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.501754    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:40.795288 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:40.795259    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"]
Nov 03 11:28:40.795288 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:28:40.795275    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4411f11b_f8a9_4a8e_824d_6319472a1963.slice/crio-f855f3482a3021a3c6cfb0c57cc6b62df1025588b5c27b08023d86491bdfe21e WatchSource:0}: Error finding container f855f3482a3021a3c6cfb0c57cc6b62df1025588b5c27b08023d86491bdfe21e: Status 404 returned error can't find the container with id f855f3482a3021a3c6cfb0c57cc6b62df1025588b5c27b08023d86491bdfe21e
Nov 03 11:28:41.130066 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:41.129995    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:41.771487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:41.771455    5817 generic.go:356] "Generic (PLEG): container finished" podID="4411f11b-f8a9-4a8e-824d-6319472a1963" containerID="1e7bc3ca0593f371b99ce3b8f33c0c68aa1a6d5e46d9cdb00362800f441dc734" exitCode=0
Nov 03 11:28:41.771637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:41.771488    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl" event={"ID":"4411f11b-f8a9-4a8e-824d-6319472a1963","Type":"ContainerDied","Data":"1e7bc3ca0593f371b99ce3b8f33c0c68aa1a6d5e46d9cdb00362800f441dc734"}
Nov 03 11:28:41.771637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:41.771540    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl" event={"ID":"4411f11b-f8a9-4a8e-824d-6319472a1963","Type":"ContainerStarted","Data":"f855f3482a3021a3c6cfb0c57cc6b62df1025588b5c27b08023d86491bdfe21e"}
Nov 03 11:28:41.776939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:41.776915    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:41.950596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:41.949953    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:42.088551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:42.088088    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"]
Nov 03 11:28:42.094048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:42.094019    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b76c62-l75zp"]
Nov 03 11:28:42.144091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:42.144062    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:42.293466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:42.293430    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:42.952642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:42.952619    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:43.103752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.103655    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-xks-ca\") pod \"4411f11b-f8a9-4a8e-824d-6319472a1963\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") "
Nov 03 11:28:43.103919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.103893    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-openshift-ca\") pod \"4411f11b-f8a9-4a8e-824d-6319472a1963\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") "
Nov 03 11:28:43.104067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.104049    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4411f11b-f8a9-4a8e-824d-6319472a1963-kube-api-access-9sr6s\" (UniqueName: \"kubernetes.io/projected/4411f11b-f8a9-4a8e-824d-6319472a1963-kube-api-access-9sr6s\") pod \"4411f11b-f8a9-4a8e-824d-6319472a1963\" (UID: \"4411f11b-f8a9-4a8e-824d-6319472a1963\") "
Nov 03 11:28:43.104290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.104265    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-openshift-ca" pod "4411f11b-f8a9-4a8e-824d-6319472a1963" (UID: "4411f11b-f8a9-4a8e-824d-6319472a1963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:43.104838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.104800    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:43.105936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.105909    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-xks-ca" pod "4411f11b-f8a9-4a8e-824d-6319472a1963" (UID: "4411f11b-f8a9-4a8e-824d-6319472a1963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:28:43.106269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.106248    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4411f11b-f8a9-4a8e-824d-6319472a1963-kube-api-access-9sr6s" pod "4411f11b-f8a9-4a8e-824d-6319472a1963" (UID: "4411f11b-f8a9-4a8e-824d-6319472a1963"). InnerVolumeSpecName "kube-api-access-9sr6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:43.206160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.206121    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sr6s\" (UniqueName: \"kubernetes.io/projected/4411f11b-f8a9-4a8e-824d-6319472a1963-kube-api-access-9sr6s\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:43.206160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.206163    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4411f11b-f8a9-4a8e-824d-6319472a1963-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:43.790922 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.790883    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl" event={"ID":"4411f11b-f8a9-4a8e-824d-6319472a1963","Type":"ContainerDied","Data":"f855f3482a3021a3c6cfb0c57cc6b62df1025588b5c27b08023d86491bdfe21e"}
Nov 03 11:28:43.791102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.790976    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f855f3482a3021a3c6cfb0c57cc6b62df1025588b5c27b08023d86491bdfe21e"
Nov 03 11:28:43.791102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.790899    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"
Nov 03 11:28:43.905731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.905688    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77435310-5f08-466a-a9ab-9d92e9bafb7f" path="/var/lib/kubelet/pods/77435310-5f08-466a-a9ab-9d92e9bafb7f/volumes"
Nov 03 11:28:43.934178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:43.934147    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:44.076358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:44.076129    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:44.973464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:44.973356    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:44.989633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:44.989106    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"]
Nov 03 11:28:44.999182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:44.997537    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"]
Nov 03 11:28:45.118987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:45.118948    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:46.016777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:46.016734    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:46.788775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:46.788361    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:47.554013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:47.551399    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:48.271887 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:48.271852    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:48.413327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:48.413238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:49.288213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:49.287994    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:49.523346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:49.523309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:49.680625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:49.680592    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:49.856086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:49.856050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:50.133590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.133486    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:50.314224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.314193    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"]
Nov 03 11:28:50.315374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.315335    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4411f11b-f8a9-4a8e-824d-6319472a1963" containerName="revoke-keys"
Nov 03 11:28:50.315374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.315367    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4411f11b-f8a9-4a8e-824d-6319472a1963" containerName="revoke-keys"
Nov 03 11:28:50.315659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.315642    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4411f11b-f8a9-4a8e-824d-6319472a1963" containerName="revoke-keys"
Nov 03 11:28:50.321455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.321433    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.326295 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.326273    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"]
Nov 03 11:28:50.341414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.341386    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:28:50.341697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.341449    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:28:50.367315 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.367287    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:50.437979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.437952    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znp85\" (UniqueName: \"kubernetes.io/projected/323311f5-d758-4a0b-bb6b-89f9afcddee9-kube-api-access-znp85\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.438097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.438068    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.438311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.438285    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.519817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.519781    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:50.544642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.544602    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.545828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.545798    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znp85\" (UniqueName: \"kubernetes.io/projected/323311f5-d758-4a0b-bb6b-89f9afcddee9-kube-api-access-znp85\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.546090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.546071    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.551621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.545623    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.570133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.570106    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.571603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.570392    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znp85\" (UniqueName: \"kubernetes.io/projected/323311f5-d758-4a0b-bb6b-89f9afcddee9-kube-api-access-znp85\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.652016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.651981    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:50.984810 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:28:50.984780    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323311f5_d758_4a0b_bb6b_89f9afcddee9.slice/crio-12398a6503dae27147f4dcff7590007310a80d19c0dfbde13665d2278fe0742c WatchSource:0}: Error finding container 12398a6503dae27147f4dcff7590007310a80d19c0dfbde13665d2278fe0742c: Status 404 returned error can't find the container with id 12398a6503dae27147f4dcff7590007310a80d19c0dfbde13665d2278fe0742c
Nov 03 11:28:50.985439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:50.985414    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"]
Nov 03 11:28:51.162537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:51.162497    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:51.864394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:51.864361    5817 generic.go:356] "Generic (PLEG): container finished" podID="323311f5-d758-4a0b-bb6b-89f9afcddee9" containerID="5014c892d5c64776bbc54743a7d4f36a0a53648786b5230a1d408b412ce545e9" exitCode=0
Nov 03 11:28:51.864793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:51.864422    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx" event={"ID":"323311f5-d758-4a0b-bb6b-89f9afcddee9","Type":"ContainerDied","Data":"5014c892d5c64776bbc54743a7d4f36a0a53648786b5230a1d408b412ce545e9"}
Nov 03 11:28:51.864793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:51.864471    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx" event={"ID":"323311f5-d758-4a0b-bb6b-89f9afcddee9","Type":"ContainerStarted","Data":"12398a6503dae27147f4dcff7590007310a80d19c0dfbde13665d2278fe0742c"}
Nov 03 11:28:51.990597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:51.989164    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:52.426295 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:28:52.420473    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:28:52.704073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:52.693694    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:53.014027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.013884    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:53.079828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.079796    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:53.225595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.221636    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-xks-ca\") pod \"323311f5-d758-4a0b-bb6b-89f9afcddee9\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") "
Nov 03 11:28:53.225595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.221717    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323311f5-d758-4a0b-bb6b-89f9afcddee9-kube-api-access-znp85\" (UniqueName: \"kubernetes.io/projected/323311f5-d758-4a0b-bb6b-89f9afcddee9-kube-api-access-znp85\") pod \"323311f5-d758-4a0b-bb6b-89f9afcddee9\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") "
Nov 03 11:28:53.225595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.221935    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-openshift-ca\") pod \"323311f5-d758-4a0b-bb6b-89f9afcddee9\" (UID: \"323311f5-d758-4a0b-bb6b-89f9afcddee9\") "
Nov 03 11:28:53.225595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.222867    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-openshift-ca" pod "323311f5-d758-4a0b-bb6b-89f9afcddee9" (UID: "323311f5-d758-4a0b-bb6b-89f9afcddee9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:53.225960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.225805    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-xks-ca" pod "323311f5-d758-4a0b-bb6b-89f9afcddee9" (UID: "323311f5-d758-4a0b-bb6b-89f9afcddee9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:28:53.228336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.228262    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323311f5-d758-4a0b-bb6b-89f9afcddee9-kube-api-access-znp85" pod "323311f5-d758-4a0b-bb6b-89f9afcddee9" (UID: "323311f5-d758-4a0b-bb6b-89f9afcddee9"). InnerVolumeSpecName "kube-api-access-znp85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:53.324320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.324233    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:53.324320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.324267    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323311f5-d758-4a0b-bb6b-89f9afcddee9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:53.324320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.324283    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znp85\" (UniqueName: \"kubernetes.io/projected/323311f5-d758-4a0b-bb6b-89f9afcddee9-kube-api-access-znp85\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:53.508602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.505249    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:53.881415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.881386    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"
Nov 03 11:28:53.881639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.881434    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx" event={"ID":"323311f5-d758-4a0b-bb6b-89f9afcddee9","Type":"ContainerDied","Data":"12398a6503dae27147f4dcff7590007310a80d19c0dfbde13665d2278fe0742c"}
Nov 03 11:28:53.881639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:53.881484    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12398a6503dae27147f4dcff7590007310a80d19c0dfbde13665d2278fe0742c"
Nov 03 11:28:54.388596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:54.378985    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:54.792680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:54.792642    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:55.152601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:55.144927    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:55.972097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:55.972053    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:56.568829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:56.568741    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:57.341595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:57.339795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:58.074923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.072550    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:58.541829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.541799    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"]
Nov 03 11:28:58.542535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.542518    5817 cpu_manager.go:414] "containerMap: removing container" podUID="323311f5-d758-4a0b-bb6b-89f9afcddee9" containerName="revoke-keys"
Nov 03 11:28:58.542629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.542537    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323311f5-d758-4a0b-bb6b-89f9afcddee9" containerName="revoke-keys"
Nov 03 11:28:58.542693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.542676    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="323311f5-d758-4a0b-bb6b-89f9afcddee9" containerName="revoke-keys"
Nov 03 11:28:58.546671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.546651    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.554613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.554542    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:28:58.554782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.554715    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"]
Nov 03 11:28:58.554782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.554734    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:28:58.635284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.635241    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.635487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.635469    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.635590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.635515    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j98l\" (UniqueName: \"kubernetes.io/projected/c6e3565d-d4a2-4515-b78d-237beff6a5cf-kube-api-access-7j98l\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.736482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.736452    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.736709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.736607    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.736709 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.736656    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j98l\" (UniqueName: \"kubernetes.io/projected/c6e3565d-d4a2-4515-b78d-237beff6a5cf-kube-api-access-7j98l\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.737700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.737674    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.739257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.739235    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.745159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.745139    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j98l\" (UniqueName: \"kubernetes.io/projected/c6e3565d-d4a2-4515-b78d-237beff6a5cf-kube-api-access-7j98l\") pod \"maas-api-revoke-sub-7fbcabd2-7w7sx\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:58.856651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.852398    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:58.874605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:58.874199    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:28:59.165730 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:28:59.165703    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e3565d_d4a2_4515_b78d_237beff6a5cf.slice/crio-8cf4d90019476bb81b438500ef11398380c87a350baa1e42ed5bcfab1d2bbb7c WatchSource:0}: Error finding container 8cf4d90019476bb81b438500ef11398380c87a350baa1e42ed5bcfab1d2bbb7c: Status 404 returned error can't find the container with id 8cf4d90019476bb81b438500ef11398380c87a350baa1e42ed5bcfab1d2bbb7c
Nov 03 11:28:59.166885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:59.166859    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"]
Nov 03 11:28:59.605294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:59.604761    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:28:59.946267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:59.946231    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx" event={"ID":"c6e3565d-d4a2-4515-b78d-237beff6a5cf","Type":"ContainerStarted","Data":"5b0bb6ed5e2b063e4c0ddad8b9c721578081c4c7511b79fcbdcb55bb2ef19f1f"}
Nov 03 11:28:59.946452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:59.946294    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx" event={"ID":"c6e3565d-d4a2-4515-b78d-237beff6a5cf","Type":"ContainerStarted","Data":"8cf4d90019476bb81b438500ef11398380c87a350baa1e42ed5bcfab1d2bbb7c"}
Nov 03 11:28:59.965389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:28:59.965347    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx" podStartSLOduration=1.965333737 podStartE2EDuration="1.965333737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:28:59.962683454 +0000 UTC m=+3570.168265505" watchObservedRunningTime="2026-11-03 11:28:59.965333737 +0000 UTC m=+3570.170916013"
Nov 03 11:29:00.024139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.024095    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" containerID="cri-o://027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc" gracePeriod=45
Nov 03 11:29:00.032648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.028632    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" containerID="cri-o://26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42" gracePeriod=45
Nov 03 11:29:00.034546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.034421    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": EOF"
Nov 03 11:29:00.090486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.090446    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": dial tcp 10.133.0.86:8000: connect: connection refused"
Nov 03 11:29:00.310625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.308876    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:00.367979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.367941    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 11:29:00.385993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.385079    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 11:29:00.477069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.476981    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-kube-api-access-rrgfm\" (UniqueName: \"kubernetes.io/projected/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-kube-api-access-rrgfm\") pod \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") "
Nov 03 11:29:00.477255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477081    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-model-cache\") pod \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") "
Nov 03 11:29:00.477255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477227    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-dshm\") pod \"4d3c3349-0d6e-4277-820c-f008f68c716d\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") "
Nov 03 11:29:00.477375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477282    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tls-certs\" (UniqueName: \"kubernetes.io/secret/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tls-certs\") pod \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") "
Nov 03 11:29:00.477375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477309    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-model-cache\") pod \"4d3c3349-0d6e-4277-820c-f008f68c716d\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") "
Nov 03 11:29:00.477505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477384    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tmp-dir\") pod \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") "
Nov 03 11:29:00.477505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477421    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-dshm\") pod \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") "
Nov 03 11:29:00.477505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477473    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3c3349-0d6e-4277-820c-f008f68c716d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3c3349-0d6e-4277-820c-f008f68c716d-tls-certs\") pod \"4d3c3349-0d6e-4277-820c-f008f68c716d\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") "
Nov 03 11:29:00.477505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477498    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-home\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-home\") pod \"4d3c3349-0d6e-4277-820c-f008f68c716d\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") "
Nov 03 11:29:00.477787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477544    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-home\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-home\") pod \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\" (UID: \"599a0d21-6f8c-433a-ab74-b9d0fc88c30a\") "
Nov 03 11:29:00.477787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477614    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3c3349-0d6e-4277-820c-f008f68c716d-kube-api-access-hmhgx\" (UniqueName: \"kubernetes.io/projected/4d3c3349-0d6e-4277-820c-f008f68c716d-kube-api-access-hmhgx\") pod \"4d3c3349-0d6e-4277-820c-f008f68c716d\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") "
Nov 03 11:29:00.477787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.477686    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-tmp-dir\") pod \"4d3c3349-0d6e-4277-820c-f008f68c716d\" (UID: \"4d3c3349-0d6e-4277-820c-f008f68c716d\") "
Nov 03 11:29:00.478120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.478042    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-model-cache" pod "4d3c3349-0d6e-4277-820c-f008f68c716d" (UID: "4d3c3349-0d6e-4277-820c-f008f68c716d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.478264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.478160    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-home" pod "4d3c3349-0d6e-4277-820c-f008f68c716d" (UID: "4d3c3349-0d6e-4277-820c-f008f68c716d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.479220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.478865    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.479220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.478891    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.480015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.479732    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-model-cache" pod "599a0d21-6f8c-433a-ab74-b9d0fc88c30a" (UID: "599a0d21-6f8c-433a-ab74-b9d0fc88c30a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.480015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.479777    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-home" pod "599a0d21-6f8c-433a-ab74-b9d0fc88c30a" (UID: "599a0d21-6f8c-433a-ab74-b9d0fc88c30a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.480015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.479796    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tmp-dir" pod "599a0d21-6f8c-433a-ab74-b9d0fc88c30a" (UID: "599a0d21-6f8c-433a-ab74-b9d0fc88c30a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.480368 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.480315    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-tmp-dir" pod "4d3c3349-0d6e-4277-820c-f008f68c716d" (UID: "4d3c3349-0d6e-4277-820c-f008f68c716d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.486068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.486042    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tls-certs" pod "599a0d21-6f8c-433a-ab74-b9d0fc88c30a" (UID: "599a0d21-6f8c-433a-ab74-b9d0fc88c30a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:00.486068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.486056    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-dshm" pod "4d3c3349-0d6e-4277-820c-f008f68c716d" (UID: "4d3c3349-0d6e-4277-820c-f008f68c716d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.486352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.486236    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-kube-api-access-rrgfm" pod "599a0d21-6f8c-433a-ab74-b9d0fc88c30a" (UID: "599a0d21-6f8c-433a-ab74-b9d0fc88c30a"). InnerVolumeSpecName "kube-api-access-rrgfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:00.486352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.485996    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-dshm" pod "599a0d21-6f8c-433a-ab74-b9d0fc88c30a" (UID: "599a0d21-6f8c-433a-ab74-b9d0fc88c30a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.488034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.488008    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3c3349-0d6e-4277-820c-f008f68c716d-tls-certs" pod "4d3c3349-0d6e-4277-820c-f008f68c716d" (UID: "4d3c3349-0d6e-4277-820c-f008f68c716d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:00.488954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.488922    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3c3349-0d6e-4277-820c-f008f68c716d-kube-api-access-hmhgx" pod "4d3c3349-0d6e-4277-820c-f008f68c716d" (UID: "4d3c3349-0d6e-4277-820c-f008f68c716d"). InnerVolumeSpecName "kube-api-access-hmhgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:00.570456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.570427    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:00.580222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580187    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3c3349-0d6e-4277-820c-f008f68c716d-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580221    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580231    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhgx\" (UniqueName: \"kubernetes.io/projected/4d3c3349-0d6e-4277-820c-f008f68c716d-kube-api-access-hmhgx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580241    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580249    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrgfm\" (UniqueName: \"kubernetes.io/projected/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-kube-api-access-rrgfm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580258    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580266    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3c3349-0d6e-4277-820c-f008f68c716d-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580274    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580283    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.580457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.580290    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599a0d21-6f8c-433a-ab74-b9d0fc88c30a-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.708621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.704446    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:00.958150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.958118    5817 generic.go:356] "Generic (PLEG): container finished" podID="c6e3565d-d4a2-4515-b78d-237beff6a5cf" containerID="5b0bb6ed5e2b063e4c0ddad8b9c721578081c4c7511b79fcbdcb55bb2ef19f1f" exitCode=0
Nov 03 11:29:00.958365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.958207    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx" event={"ID":"c6e3565d-d4a2-4515-b78d-237beff6a5cf","Type":"ContainerDied","Data":"5b0bb6ed5e2b063e4c0ddad8b9c721578081c4c7511b79fcbdcb55bb2ef19f1f"}
Nov 03 11:29:00.959656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.959622    5817 generic.go:356] "Generic (PLEG): container finished" podID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerID="26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42" exitCode=0
Nov 03 11:29:00.959783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.959668    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"
Nov 03 11:29:00.959783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.959704    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" event={"ID":"4d3c3349-0d6e-4277-820c-f008f68c716d","Type":"ContainerDied","Data":"26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42"}
Nov 03 11:29:00.959783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.959744    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq" event={"ID":"4d3c3349-0d6e-4277-820c-f008f68c716d","Type":"ContainerDied","Data":"5e8eea1fc1f1fdb0207b073400f88d70b6e6046e8f5724f3b30249a128082e42"}
Nov 03 11:29:00.959783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.959772    5817 scope.go:122] "RemoveContainer" containerID="26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42"
Nov 03 11:29:00.961444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.961421    5817 generic.go:356] "Generic (PLEG): container finished" podID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerID="027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc" exitCode=0
Nov 03 11:29:00.961617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.961460    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"
Nov 03 11:29:00.961617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.961488    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" event={"ID":"599a0d21-6f8c-433a-ab74-b9d0fc88c30a","Type":"ContainerDied","Data":"027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc"}
Nov 03 11:29:00.961617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.961513    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj" event={"ID":"599a0d21-6f8c-433a-ab74-b9d0fc88c30a","Type":"ContainerDied","Data":"5ea76b88affc9219064ed41ab7daf7f67fbbbc7be0915b5a01bd81cbfc47e8a1"}
Nov 03 11:29:00.975389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.975366    5817 scope.go:122] "RemoveContainer" containerID="26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42"
Nov 03 11:29:00.975823 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:00.975795    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42\": container with ID starting with 26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42 not found: ID does not exist" containerID="26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42"
Nov 03 11:29:00.975929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.975835    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42"} err="failed to get container status \"26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42\": rpc error: code = NotFound desc = could not find container \"26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42\": container with ID starting with 26b7a8c180b3e1f9b8bac5646147676a51166ccc21f90be04a856983f7a88e42 not found: ID does not exist"
Nov 03 11:29:00.975929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.975862    5817 scope.go:122] "RemoveContainer" containerID="027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc"
Nov 03 11:29:00.994750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.994562    5817 scope.go:122] "RemoveContainer" containerID="027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc"
Nov 03 11:29:00.995213 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:00.995191    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc\": container with ID starting with 027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc not found: ID does not exist" containerID="027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc"
Nov 03 11:29:00.995314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:00.995220    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc"} err="failed to get container status \"027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc\": rpc error: code = NotFound desc = could not find container \"027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc\": container with ID starting with 027c2262347312d249bd68b00e24fa2295092a488312241ae114cd313c99bdcc not found: ID does not exist"
Nov 03 11:29:01.012656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.010632    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"]
Nov 03 11:29:01.015585 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.015023    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/premium-w2-e78c33-kserve-57b77754df-xcpxq"]
Nov 03 11:29:01.020424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.020395    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"]
Nov 03 11:29:01.024085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.024062    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-2-w2-e78c33-kserve-746fb46fc9-s4gjj"]
Nov 03 11:29:01.115047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.114645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:01.762405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.762370    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:29:01.911767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.911732    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" path="/var/lib/kubelet/pods/4d3c3349-0d6e-4277-820c-f008f68c716d/volumes"
Nov 03 11:29:01.912460 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.912437    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" path="/var/lib/kubelet/pods/599a0d21-6f8c-433a-ab74-b9d0fc88c30a/volumes"
Nov 03 11:29:01.987595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:01.984363    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:02.176096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.176072    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:29:02.333396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.333315    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e3565d-d4a2-4515-b78d-237beff6a5cf-kube-api-access-7j98l\" (UniqueName: \"kubernetes.io/projected/c6e3565d-d4a2-4515-b78d-237beff6a5cf-kube-api-access-7j98l\") pod \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") "
Nov 03 11:29:02.333560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.333428    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-openshift-ca\") pod \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") "
Nov 03 11:29:02.333560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.333503    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-xks-ca\") pod \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\" (UID: \"c6e3565d-d4a2-4515-b78d-237beff6a5cf\") "
Nov 03 11:29:02.333880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.333854    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-openshift-ca" pod "c6e3565d-d4a2-4515-b78d-237beff6a5cf" (UID: "c6e3565d-d4a2-4515-b78d-237beff6a5cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:02.334217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.334128    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:02.335716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.335686    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e3565d-d4a2-4515-b78d-237beff6a5cf-kube-api-access-7j98l" pod "c6e3565d-d4a2-4515-b78d-237beff6a5cf" (UID: "c6e3565d-d4a2-4515-b78d-237beff6a5cf"). InnerVolumeSpecName "kube-api-access-7j98l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:02.335848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.335789    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-xks-ca" pod "c6e3565d-d4a2-4515-b78d-237beff6a5cf" (UID: "c6e3565d-d4a2-4515-b78d-237beff6a5cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:02.435502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.435472    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j98l\" (UniqueName: \"kubernetes.io/projected/c6e3565d-d4a2-4515-b78d-237beff6a5cf-kube-api-access-7j98l\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:02.435502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.435500    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e3565d-d4a2-4515-b78d-237beff6a5cf-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:02.907049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:02.907010    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:03.078305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:03.076193    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx" event={"ID":"c6e3565d-d4a2-4515-b78d-237beff6a5cf","Type":"ContainerDied","Data":"8cf4d90019476bb81b438500ef11398380c87a350baa1e42ed5bcfab1d2bbb7c"}
Nov 03 11:29:03.078631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:03.078600    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cf4d90019476bb81b438500ef11398380c87a350baa1e42ed5bcfab1d2bbb7c"
Nov 03 11:29:03.078929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:03.078914    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"
Nov 03 11:29:03.775414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:03.775330    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:04.398629 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:04.388539    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:04.539935 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:04.539893    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:04.691544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:04.691512    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:05.444688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:05.442523    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:06.140600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:06.135414    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:06.271927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:06.271891    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:06.780957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:06.780922    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:07.462505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:07.462472    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:07.614045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:07.614000    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:08.277269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:08.277230    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:08.745009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:08.744969    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:08.888365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:08.888334    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:09.575656 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:09.568225    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:10.196479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:10.196450    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:10.353608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:10.353012    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:10.489069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:10.488980    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"]
Nov 03 11:29:10.500378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:10.499891    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:10.500989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:10.500954    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e92a08-lxcxx"]
Nov 03 11:29:10.725592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:10.725533    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:11.026302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:11.026267    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:11.616597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:11.614488    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:11.903242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:11.903169    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323311f5-d758-4a0b-bb6b-89f9afcddee9" path="/var/lib/kubelet/pods/323311f5-d758-4a0b-bb6b-89f9afcddee9/volumes"
Nov 03 11:29:12.011434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.011390    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"]
Nov 03 11:29:12.012414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012390    5817 cpu_manager.go:414] "containerMap: removing container" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main"
Nov 03 11:29:12.012414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012414    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main"
Nov 03 11:29:12.012645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012430    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main"
Nov 03 11:29:12.012645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012438    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main"
Nov 03 11:29:12.012645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012463    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c6e3565d-d4a2-4515-b78d-237beff6a5cf" containerName="revoke-keys"
Nov 03 11:29:12.012645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012471    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e3565d-d4a2-4515-b78d-237beff6a5cf" containerName="revoke-keys"
Nov 03 11:29:12.012874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012691    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="599a0d21-6f8c-433a-ab74-b9d0fc88c30a" containerName="main"
Nov 03 11:29:12.012874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012704    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3c3349-0d6e-4277-820c-f008f68c716d" containerName="main"
Nov 03 11:29:12.012874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.012717    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e3565d-d4a2-4515-b78d-237beff6a5cf" containerName="revoke-keys"
Nov 03 11:29:12.019604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.018805    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.025427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.025395    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"]
Nov 03 11:29:12.039366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.039342    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:29:12.040110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.040085    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:29:12.120336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.120309    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.120493 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.120365    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.120563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.120547    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjlqf\" (UniqueName: \"kubernetes.io/projected/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-kube-api-access-jjlqf\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.221586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.221537    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjlqf\" (UniqueName: \"kubernetes.io/projected/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-kube-api-access-jjlqf\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.221787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.221688    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.221787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.221755    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.222399 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.222376    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.224327 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.224304    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.230861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.230837    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjlqf\" (UniqueName: \"kubernetes.io/projected/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-kube-api-access-jjlqf\") pod \"maas-api-revoke-sub-72db4f18-nwhfg\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.363596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.361834    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:12.380253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.378227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:12.685295 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:29:12.685265    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62da97d_bfc8_4a9a_94c4_d87895bdcf5e.slice/crio-042c87f27b5357c8f83dbf763f206e8702483e20df53f84d13f6e201607438e9 WatchSource:0}: Error finding container 042c87f27b5357c8f83dbf763f206e8702483e20df53f84d13f6e201607438e9: Status 404 returned error can't find the container with id 042c87f27b5357c8f83dbf763f206e8702483e20df53f84d13f6e201607438e9
Nov 03 11:29:12.685535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.685514    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"]
Nov 03 11:29:12.698715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:12.698687    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:13.168941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:13.168906    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg" event={"ID":"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e","Type":"ContainerStarted","Data":"7582f3e91393159d7f753447e28e57b6008d3f05b3988ee38c0f2992d1a77249"}
Nov 03 11:29:13.169359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:13.168960    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg" event={"ID":"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e","Type":"ContainerStarted","Data":"042c87f27b5357c8f83dbf763f206e8702483e20df53f84d13f6e201607438e9"}
Nov 03 11:29:13.171197 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:13.171018    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:13.190788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:13.190735    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg" podStartSLOduration=2.190717704 podStartE2EDuration="2.190717704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:13.18627874 +0000 UTC m=+3583.391860792" watchObservedRunningTime="2026-11-03 11:29:13.190717704 +0000 UTC m=+3583.396299756"
Nov 03 11:29:13.745100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:13.745064    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:14.178850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:14.178819    5817 generic.go:356] "Generic (PLEG): container finished" podID="e62da97d-bfc8-4a9a-94c4-d87895bdcf5e" containerID="7582f3e91393159d7f753447e28e57b6008d3f05b3988ee38c0f2992d1a77249" exitCode=0
Nov 03 11:29:14.179332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:14.178943    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg" event={"ID":"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e","Type":"ContainerDied","Data":"7582f3e91393159d7f753447e28e57b6008d3f05b3988ee38c0f2992d1a77249"}
Nov 03 11:29:14.471873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:14.461604    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:15.182598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.173236    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:15.359371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.359347    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:15.497586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.497472    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-xks-ca\") pod \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") "
Nov 03 11:29:15.497586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.497538    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-openshift-ca\") pod \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") "
Nov 03 11:29:15.497841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.497641    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-kube-api-access-jjlqf\" (UniqueName: \"kubernetes.io/projected/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-kube-api-access-jjlqf\") pod \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\" (UID: \"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e\") "
Nov 03 11:29:15.497933 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.497911    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-openshift-ca" pod "e62da97d-bfc8-4a9a-94c4-d87895bdcf5e" (UID: "e62da97d-bfc8-4a9a-94c4-d87895bdcf5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:15.498402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.498367    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:15.499895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.499868    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-xks-ca" pod "e62da97d-bfc8-4a9a-94c4-d87895bdcf5e" (UID: "e62da97d-bfc8-4a9a-94c4-d87895bdcf5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:15.500337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.500311    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-kube-api-access-jjlqf" pod "e62da97d-bfc8-4a9a-94c4-d87895bdcf5e" (UID: "e62da97d-bfc8-4a9a-94c4-d87895bdcf5e"). InnerVolumeSpecName "kube-api-access-jjlqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:15.599672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.599638    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:15.599672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.599671    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjlqf\" (UniqueName: \"kubernetes.io/projected/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e-kube-api-access-jjlqf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:15.895662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:15.891713    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:16.200458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:16.200429    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"
Nov 03 11:29:16.200905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:16.200458    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg" event={"ID":"e62da97d-bfc8-4a9a-94c4-d87895bdcf5e","Type":"ContainerDied","Data":"042c87f27b5357c8f83dbf763f206e8702483e20df53f84d13f6e201607438e9"}
Nov 03 11:29:16.200905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:16.200502    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="042c87f27b5357c8f83dbf763f206e8702483e20df53f84d13f6e201607438e9"
Nov 03 11:29:16.706424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:16.706383    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:17.352254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:17.352218    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:17.516027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:17.515993    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:17.657854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:17.657770    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"]
Nov 03 11:29:17.665130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:17.665100    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:17.673077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:17.673044    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e78c33-q9bcl"]
Nov 03 11:29:17.815131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:17.815075    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:17.903943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:17.903909    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4411f11b-f8a9-4a8e-824d-6319472a1963" path="/var/lib/kubelet/pods/4411f11b-f8a9-4a8e-824d-6319472a1963/volumes"
Nov 03 11:29:17.969627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:17.969592    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:18.199111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.199077    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:18.349627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.349518    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:18.731294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.731257    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"]
Nov 03 11:29:18.732386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.732361    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e62da97d-bfc8-4a9a-94c4-d87895bdcf5e" containerName="revoke-keys"
Nov 03 11:29:18.732553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.732386    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62da97d-bfc8-4a9a-94c4-d87895bdcf5e" containerName="revoke-keys"
Nov 03 11:29:18.732686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.732646    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62da97d-bfc8-4a9a-94c4-d87895bdcf5e" containerName="revoke-keys"
Nov 03 11:29:18.737660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.737636    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.745158 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.745135    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"]
Nov 03 11:29:18.773822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.773799    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:29:18.773984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.773830    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:29:18.877598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.876599    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.877598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.876657    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2p7g\" (UniqueName: \"kubernetes.io/projected/9359fe7c-e311-45cf-87b2-4f7204d8234f-kube-api-access-d2p7g\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.877598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.876773    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.882596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.879607    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:18.978342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.978304    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.978551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.978348    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2p7g\" (UniqueName: \"kubernetes.io/projected/9359fe7c-e311-45cf-87b2-4f7204d8234f-kube-api-access-d2p7g\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.978551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.978433    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.979048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.979013    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.981085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.981059    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:18.987412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:18.987353    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2p7g\" (UniqueName: \"kubernetes.io/projected/9359fe7c-e311-45cf-87b2-4f7204d8234f-kube-api-access-d2p7g\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:19.058836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:19.058808    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:19.263711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:19.263618    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:19.348091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:19.348062    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"]
Nov 03 11:29:19.349675 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:29:19.349651    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9359fe7c_e311_45cf_87b2_4f7204d8234f.slice/crio-9ca31d0ecbf06f6d9f49010005bdfa289cf2da29ee1b2c8c756fb8a760cfcbeb WatchSource:0}: Error finding container 9ca31d0ecbf06f6d9f49010005bdfa289cf2da29ee1b2c8c756fb8a760cfcbeb: Status 404 returned error can't find the container with id 9ca31d0ecbf06f6d9f49010005bdfa289cf2da29ee1b2c8c756fb8a760cfcbeb
Nov 03 11:29:19.468237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:19.468199    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:19.870551 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:19.870464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:20.242969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:20.242935    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w" event={"ID":"9359fe7c-e311-45cf-87b2-4f7204d8234f","Type":"ContainerStarted","Data":"792dfd0453877722da1674e611e0c6e71316fffa24f0f5cea086ec89375c5025"}
Nov 03 11:29:20.243199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:20.243004    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w" event={"ID":"9359fe7c-e311-45cf-87b2-4f7204d8234f","Type":"ContainerStarted","Data":"9ca31d0ecbf06f6d9f49010005bdfa289cf2da29ee1b2c8c756fb8a760cfcbeb"}
Nov 03 11:29:20.263546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:20.263498    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w" podStartSLOduration=2.263484621 podStartE2EDuration="2.263484621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:20.258332899 +0000 UTC m=+3590.463914950" watchObservedRunningTime="2026-11-03 11:29:20.263484621 +0000 UTC m=+3590.469066672"
Nov 03 11:29:20.458377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:20.456439    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:21.234233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:21.234203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:21.254464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:21.254436    5817 generic.go:356] "Generic (PLEG): container finished" podID="9359fe7c-e311-45cf-87b2-4f7204d8234f" containerID="792dfd0453877722da1674e611e0c6e71316fffa24f0f5cea086ec89375c5025" exitCode=0
Nov 03 11:29:21.254655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:21.254504    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w" event={"ID":"9359fe7c-e311-45cf-87b2-4f7204d8234f","Type":"ContainerDied","Data":"792dfd0453877722da1674e611e0c6e71316fffa24f0f5cea086ec89375c5025"}
Nov 03 11:29:21.975317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:21.974614    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:22.438170 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:22.438133    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:29:22.450770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.450746    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:22.534627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.534531    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-xks-ca\") pod \"9359fe7c-e311-45cf-87b2-4f7204d8234f\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") "
Nov 03 11:29:22.534627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.534597    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-openshift-ca\") pod \"9359fe7c-e311-45cf-87b2-4f7204d8234f\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") "
Nov 03 11:29:22.534831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.534816    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9359fe7c-e311-45cf-87b2-4f7204d8234f-kube-api-access-d2p7g\" (UniqueName: \"kubernetes.io/projected/9359fe7c-e311-45cf-87b2-4f7204d8234f-kube-api-access-d2p7g\") pod \"9359fe7c-e311-45cf-87b2-4f7204d8234f\" (UID: \"9359fe7c-e311-45cf-87b2-4f7204d8234f\") "
Nov 03 11:29:22.534984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.534952    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-openshift-ca" pod "9359fe7c-e311-45cf-87b2-4f7204d8234f" (UID: "9359fe7c-e311-45cf-87b2-4f7204d8234f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:22.535526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.535444    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:22.536963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.536934    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-xks-ca" pod "9359fe7c-e311-45cf-87b2-4f7204d8234f" (UID: "9359fe7c-e311-45cf-87b2-4f7204d8234f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:22.537091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.537049    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9359fe7c-e311-45cf-87b2-4f7204d8234f-kube-api-access-d2p7g" pod "9359fe7c-e311-45cf-87b2-4f7204d8234f" (UID: "9359fe7c-e311-45cf-87b2-4f7204d8234f"). InnerVolumeSpecName "kube-api-access-d2p7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:22.642663 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.642597    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9359fe7c-e311-45cf-87b2-4f7204d8234f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:22.643112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.643087    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2p7g\" (UniqueName: \"kubernetes.io/projected/9359fe7c-e311-45cf-87b2-4f7204d8234f-kube-api-access-d2p7g\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:22.656208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:22.655045    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:23.280396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:23.280227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:23.321596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:23.319231    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w" event={"ID":"9359fe7c-e311-45cf-87b2-4f7204d8234f","Type":"ContainerDied","Data":"9ca31d0ecbf06f6d9f49010005bdfa289cf2da29ee1b2c8c756fb8a760cfcbeb"}
Nov 03 11:29:23.321596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:23.319297    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ca31d0ecbf06f6d9f49010005bdfa289cf2da29ee1b2c8c756fb8a760cfcbeb"
Nov 03 11:29:23.321596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:23.319375    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"
Nov 03 11:29:23.961021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:23.960987    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:24.028873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:24.028832    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"]
Nov 03 11:29:24.035989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:24.035949    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f006aed1-dxntd"]
Nov 03 11:29:24.096772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:24.096736    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:24.664204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:24.664170    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:24.832397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:24.832366    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:24.840879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:24.840854    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"]
Nov 03 11:29:24.841111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:24.841091    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy" containerID="cri-o://bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da" gracePeriod=30
Nov 03 11:29:25.117355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:25.117326    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:25.462097 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:25.462061    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:25.904634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:25.904546    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a31bce-9447-4ee9-b1f5-bb74ac93834e" path="/var/lib/kubelet/pods/09a31bce-9447-4ee9-b1f5-bb74ac93834e/volumes"
Nov 03 11:29:26.094361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:26.091800    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:26.826116 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:26.821611    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:27.582619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:27.582565    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:28.352598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:28.347854    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:28.835302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:28.835260    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"]
Nov 03 11:29:28.936154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:28.936119    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:29.106700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:29.100458    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:29.747594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:29.745194    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:30.144867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.144839    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 11:29:30.286748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.286652    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-credential-socket\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.286929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.286746    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-podinfo\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.286929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.286806    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-data\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.286929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.286867    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-certs\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.287126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.286927    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-credential-socket" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:30.287126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.286957    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-envoy\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.287126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.287016    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-kube-api-access-vt2vq\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-kube-api-access-vt2vq\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.287126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.287046    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34d98ad-5d1e-4591-95f9-551ecedda34d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d98ad-5d1e-4591-95f9-551ecedda34d-istiod-ca-cert\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.287126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.287084    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-socket\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.287126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.287097    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-data" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:30.287477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.287131    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-token\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-token\") pod \"e34d98ad-5d1e-4591-95f9-551ecedda34d\" (UID: \"e34d98ad-5d1e-4591-95f9-551ecedda34d\") "
Nov 03 11:29:30.287477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.287188    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-certs" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:30.287644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.287519    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34d98ad-5d1e-4591-95f9-551ecedda34d-istiod-ca-cert" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:30.287644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.287545    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-socket" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:30.288156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.288131    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d98ad-5d1e-4591-95f9-551ecedda34d-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.288339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.288315    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.288467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.288342    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.288467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.288356    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.288467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.288369    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.289747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.289722    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-envoy" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:30.289996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.289976    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-token" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:30.290071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.290037    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-kube-api-access-vt2vq" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "kube-api-access-vt2vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:30.290144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.290103    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-podinfo" pod "e34d98ad-5d1e-4591-95f9-551ecedda34d" (UID: "e34d98ad-5d1e-4591-95f9-551ecedda34d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:29:30.390067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.390035    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.390067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.390064    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt2vq\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-kube-api-access-vt2vq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.390067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.390075    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.390309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.390086    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d98ad-5d1e-4591-95f9-551ecedda34d-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:30.399247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.399216    5817 generic.go:356] "Generic (PLEG): container finished" podID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerID="bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da" exitCode=0
Nov 03 11:29:30.399386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.399277    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"
Nov 03 11:29:30.399386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.399279    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" event={"ID":"e34d98ad-5d1e-4591-95f9-551ecedda34d","Type":"ContainerDied","Data":"bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da"}
Nov 03 11:29:30.399468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.399414    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd" event={"ID":"e34d98ad-5d1e-4591-95f9-551ecedda34d","Type":"ContainerDied","Data":"3236ae54a3adcbf8e8ac519856ee0160bed919555e1f1adb9d1b548eb37ffd65"}
Nov 03 11:29:30.399468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.399442    5817 scope.go:122] "RemoveContainer" containerID="bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da"
Nov 03 11:29:30.412565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.412546    5817 scope.go:122] "RemoveContainer" containerID="bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da"
Nov 03 11:29:30.412881 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:30.412847    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da\": container with ID starting with bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da not found: ID does not exist" containerID="bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da"
Nov 03 11:29:30.412987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.412888    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da"} err="failed to get container status \"bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da\": rpc error: code = NotFound desc = could not find container \"bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da\": container with ID starting with bb49d346e72f5a4541f04c6c3df7029d5e651fc2a7dfafbb753ef635309cf7da not found: ID does not exist"
Nov 03 11:29:30.434053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.434025    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"]
Nov 03 11:29:30.439564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.439542    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-v24hd"]
Nov 03 11:29:30.511519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:30.511481    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:31.632193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:31.632154    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:31.903008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:31.902926    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" path="/var/lib/kubelet/pods/e34d98ad-5d1e-4591-95f9-551ecedda34d/volumes"
Nov 03 11:29:32.492833 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.492796    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:32.846834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.846729    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"]
Nov 03 11:29:32.847398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.847383    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9359fe7c-e311-45cf-87b2-4f7204d8234f" containerName="revoke-keys"
Nov 03 11:29:32.847398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.847398    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9359fe7c-e311-45cf-87b2-4f7204d8234f" containerName="revoke-keys"
Nov 03 11:29:32.847524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.847412    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy"
Nov 03 11:29:32.847524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.847417    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy"
Nov 03 11:29:32.847524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.847501    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34d98ad-5d1e-4591-95f9-551ecedda34d" containerName="istio-proxy"
Nov 03 11:29:32.847524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.847520    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9359fe7c-e311-45cf-87b2-4f7204d8234f" containerName="revoke-keys"
Nov 03 11:29:32.850984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.850964    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:32.855558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.855538    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:29:32.855895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.855850    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:29:32.858497 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.858473    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"]
Nov 03 11:29:32.943718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.943688    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:32.943867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.943807    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:32.943930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:32.943890    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6w8b\" (UniqueName: \"kubernetes.io/projected/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-kube-api-access-m6w8b\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:33.045723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.045691    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6w8b\" (UniqueName: \"kubernetes.io/projected/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-kube-api-access-m6w8b\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:33.045925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.045871    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:33.046011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.045991    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:33.046622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.046565    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:33.048561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.048538    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:33.053017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.052996    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6w8b\" (UniqueName: \"kubernetes.io/projected/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-kube-api-access-m6w8b\") pod \"maas-api-revoke-sub-f4179e8e-ng88v\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:33.171047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.170973    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:33.390605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.389906    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:33.455260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.455231    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"]
Nov 03 11:29:33.456689 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:29:33.456662    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d617dc3_d6b3_4292_87a2_b8e6ff353ed1.slice/crio-343ccfbd8453b087d809dc1cf1576a89f8b519bdf1c3c870544e435165991bc9 WatchSource:0}: Error finding container 343ccfbd8453b087d809dc1cf1576a89f8b519bdf1c3c870544e435165991bc9: Status 404 returned error can't find the container with id 343ccfbd8453b087d809dc1cf1576a89f8b519bdf1c3c870544e435165991bc9
Nov 03 11:29:33.484340 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.484313    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v" event={"ID":"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1","Type":"ContainerStarted","Data":"343ccfbd8453b087d809dc1cf1576a89f8b519bdf1c3c870544e435165991bc9"}
Nov 03 11:29:33.836643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:33.836611    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw" podUID="33600aaa-9f93-471c-951b-54362ed05f01" containerName="payload-pre-processing" containerID="cri-o://93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317" gracePeriod=25
Nov 03 11:29:34.083508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:34.083403    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:34.425960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:34.425922    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw" podUID="33600aaa-9f93-471c-951b-54362ed05f01" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:29:34.496489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:34.496455    5817 generic.go:356] "Generic (PLEG): container finished" podID="8d617dc3-d6b3-4292-87a2-b8e6ff353ed1" containerID="b3c6dd6ec053294a402cc4a5e4dc61c753cd3a8dc74346bb89eae3878c83d93e" exitCode=0
Nov 03 11:29:34.496673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:34.496546    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v" event={"ID":"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1","Type":"ContainerDied","Data":"b3c6dd6ec053294a402cc4a5e4dc61c753cd3a8dc74346bb89eae3878c83d93e"}
Nov 03 11:29:34.787678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:34.787646    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:35.399773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.398645    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:35.677674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.677650    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:35.719259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.719224    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-kube-api-access-m6w8b\" (UniqueName: \"kubernetes.io/projected/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-kube-api-access-m6w8b\") pod \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") "
Nov 03 11:29:35.719410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.719322    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-xks-ca\") pod \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") "
Nov 03 11:29:35.719500 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.719478    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-openshift-ca\") pod \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\" (UID: \"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1\") "
Nov 03 11:29:35.720363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.720335    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-openshift-ca" pod "8d617dc3-d6b3-4292-87a2-b8e6ff353ed1" (UID: "8d617dc3-d6b3-4292-87a2-b8e6ff353ed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:35.722533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.722512    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-kube-api-access-m6w8b" pod "8d617dc3-d6b3-4292-87a2-b8e6ff353ed1" (UID: "8d617dc3-d6b3-4292-87a2-b8e6ff353ed1"). InnerVolumeSpecName "kube-api-access-m6w8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:35.722843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.722817    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-xks-ca" pod "8d617dc3-d6b3-4292-87a2-b8e6ff353ed1" (UID: "8d617dc3-d6b3-4292-87a2-b8e6ff353ed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:35.811037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.811006    5817 scope.go:122] "RemoveContainer" containerID="55d24b5a221ec682f0c811d95b602cda200d2446f209da877454889415f93bfb"
Nov 03 11:29:35.826846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.826813    5817 scope.go:122] "RemoveContainer" containerID="a208708dba75885734d79033cc360afba1fec576cf42bbbba093e1b596ad7910"
Nov 03 11:29:35.830316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.829647    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:35.830316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.829674    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:35.830316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.829693    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6w8b\" (UniqueName: \"kubernetes.io/projected/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1-kube-api-access-m6w8b\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:35.841651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.841630    5817 scope.go:122] "RemoveContainer" containerID="7af377584495a5eaafad930370863ed77d5875039a326d09a24181904f9cd26a"
Nov 03 11:29:35.885438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:35.885411    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:36.100610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.099237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:36.206428 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.206392    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"]
Nov 03 11:29:36.210154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.210127    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-fcc2ce-xhd9w"]
Nov 03 11:29:36.273675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.273643    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:36.425622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.424957    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:36.521339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.521254    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"
Nov 03 11:29:36.521339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.521293    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v" event={"ID":"8d617dc3-d6b3-4292-87a2-b8e6ff353ed1","Type":"ContainerDied","Data":"343ccfbd8453b087d809dc1cf1576a89f8b519bdf1c3c870544e435165991bc9"}
Nov 03 11:29:36.521566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.521346    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="343ccfbd8453b087d809dc1cf1576a89f8b519bdf1c3c870544e435165991bc9"
Nov 03 11:29:36.831264 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:36.831183    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:37.160635 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:37.153695    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:37.780354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:37.777736    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:37.904131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:37.904098    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9359fe7c-e311-45cf-87b2-4f7204d8234f" path="/var/lib/kubelet/pods/9359fe7c-e311-45cf-87b2-4f7204d8234f/volumes"
Nov 03 11:29:38.384494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:38.384460    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:38.973653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:38.973618    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:39.576876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:39.576776    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:40.253682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:40.253528    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:40.881359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:40.881327    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:41.655655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:41.655619    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:41.855788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:41.855749    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:42.460532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:42.460502    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:42.614725 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:42.614690    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:43.260797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.260646    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:43.293996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.293966    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"]
Nov 03 11:29:43.295250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.295223    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8d617dc3-d6b3-4292-87a2-b8e6ff353ed1" containerName="revoke-keys"
Nov 03 11:29:43.295441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.295425    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d617dc3-d6b3-4292-87a2-b8e6ff353ed1" containerName="revoke-keys"
Nov 03 11:29:43.295752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.295729    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d617dc3-d6b3-4292-87a2-b8e6ff353ed1" containerName="revoke-keys"
Nov 03 11:29:43.303682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.303659    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"]
Nov 03 11:29:43.303855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.303773    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.309787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.309730    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:29:43.310441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.310394    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:29:43.357592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.357541    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.357795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.357775    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.357868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.357830    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdlv\" (UniqueName: \"kubernetes.io/projected/f84c9456-bce4-4d4b-aea1-43680d02755f-kube-api-access-sjdlv\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.459345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.459301    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdlv\" (UniqueName: \"kubernetes.io/projected/f84c9456-bce4-4d4b-aea1-43680d02755f-kube-api-access-sjdlv\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.459492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.459450    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.459691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.459673    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.460275 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.460249    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.462033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.462012    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.467926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.467896    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdlv\" (UniqueName: \"kubernetes.io/projected/f84c9456-bce4-4d4b-aea1-43680d02755f-kube-api-access-sjdlv\") pod \"maas-api-revoke-sub-15242534-q2x5l\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.548711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.548613    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:43.640597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.639911    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:43.712293 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.712259    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:43.851365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.851277    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"]
Nov 03 11:29:43.852196 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:43.851849    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy" containerID="cri-o://006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978" gracePeriod=30
Nov 03 11:29:44.021596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:44.005088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:44.109713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:44.109666    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"]
Nov 03 11:29:44.114925 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:29:44.114889    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84c9456_bce4_4d4b_aea1_43680d02755f.slice/crio-89467bc9296c0bfa739ec556bd4f9d676a58158b0112eeac3e99769324ec71cb WatchSource:0}: Error finding container 89467bc9296c0bfa739ec556bd4f9d676a58158b0112eeac3e99769324ec71cb: Status 404 returned error can't find the container with id 89467bc9296c0bfa739ec556bd4f9d676a58158b0112eeac3e99769324ec71cb
Nov 03 11:29:44.275879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:44.274302    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:44.426142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:44.426106    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw" podUID="33600aaa-9f93-471c-951b-54362ed05f01" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:29:44.464047 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:44.464014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:44.609525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:44.609485    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l" event={"ID":"f84c9456-bce4-4d4b-aea1-43680d02755f","Type":"ContainerStarted","Data":"9634a093962f981cf3b1612e71f4e2b3a5d6d69000056933a7e617cb4eb9e71a"}
Nov 03 11:29:44.609771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:44.609600    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l" event={"ID":"f84c9456-bce4-4d4b-aea1-43680d02755f","Type":"ContainerStarted","Data":"89467bc9296c0bfa739ec556bd4f9d676a58158b0112eeac3e99769324ec71cb"}
Nov 03 11:29:44.629040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:44.628993    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l" podStartSLOduration=1.628979234 podStartE2EDuration="1.628979234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:44.626836487 +0000 UTC m=+3614.832418539" watchObservedRunningTime="2026-11-03 11:29:44.628979234 +0000 UTC m=+3614.834561286"
Nov 03 11:29:45.328984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:45.328902    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:45.619834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:45.619742    5817 generic.go:356] "Generic (PLEG): container finished" podID="f84c9456-bce4-4d4b-aea1-43680d02755f" containerID="9634a093962f981cf3b1612e71f4e2b3a5d6d69000056933a7e617cb4eb9e71a" exitCode=0
Nov 03 11:29:45.619834 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:45.619792    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l" event={"ID":"f84c9456-bce4-4d4b-aea1-43680d02755f","Type":"ContainerDied","Data":"9634a093962f981cf3b1612e71f4e2b3a5d6d69000056933a7e617cb4eb9e71a"}
Nov 03 11:29:46.091615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.080886    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:46.830597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.828862    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:46.842416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.842387    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:46.939484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.939459    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-xks-ca\") pod \"f84c9456-bce4-4d4b-aea1-43680d02755f\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") "
Nov 03 11:29:46.939679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.939584    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84c9456-bce4-4d4b-aea1-43680d02755f-kube-api-access-sjdlv\" (UniqueName: \"kubernetes.io/projected/f84c9456-bce4-4d4b-aea1-43680d02755f-kube-api-access-sjdlv\") pod \"f84c9456-bce4-4d4b-aea1-43680d02755f\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") "
Nov 03 11:29:46.939777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.939697    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-openshift-ca\") pod \"f84c9456-bce4-4d4b-aea1-43680d02755f\" (UID: \"f84c9456-bce4-4d4b-aea1-43680d02755f\") "
Nov 03 11:29:46.940102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.940076    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-openshift-ca" pod "f84c9456-bce4-4d4b-aea1-43680d02755f" (UID: "f84c9456-bce4-4d4b-aea1-43680d02755f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:46.940619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.940561    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:46.941957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.941929    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84c9456-bce4-4d4b-aea1-43680d02755f-kube-api-access-sjdlv" pod "f84c9456-bce4-4d4b-aea1-43680d02755f" (UID: "f84c9456-bce4-4d4b-aea1-43680d02755f"). InnerVolumeSpecName "kube-api-access-sjdlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:46.942141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:46.942124    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-xks-ca" pod "f84c9456-bce4-4d4b-aea1-43680d02755f" (UID: "f84c9456-bce4-4d4b-aea1-43680d02755f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:46.942718 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:46.942696    5817 token_manager.go:124] "Couldn't update token" err="serviceaccounts \"default\" is forbidden: unable to create new content in namespace e2e-models-e2e-worker-w4-d5d3c3 because it is being terminated" cacheKey="\"default\"/\"e2e-models-e2e-worker-w4-d5d3c3\"/[]string(nil)/3607/v1.BoundObjectReference{Kind:\"Pod\", APIVersion:\"v1\", Name:\"premium-w4-d5d3c3-kserve-55cbb58654-j6hkp\", UID:\"10f80020-b416-45e1-b616-ee44e0668e45\"}/\"\""
Nov 03 11:29:47.042104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:47.042071    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c9456-bce4-4d4b-aea1-43680d02755f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:47.042104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:47.042106    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjdlv\" (UniqueName: \"kubernetes.io/projected/f84c9456-bce4-4d4b-aea1-43680d02755f-kube-api-access-sjdlv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:47.572817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:47.572768    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:47.640234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:47.640189    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l" event={"ID":"f84c9456-bce4-4d4b-aea1-43680d02755f","Type":"ContainerDied","Data":"89467bc9296c0bfa739ec556bd4f9d676a58158b0112eeac3e99769324ec71cb"}
Nov 03 11:29:47.640388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:47.640255    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89467bc9296c0bfa739ec556bd4f9d676a58158b0112eeac3e99769324ec71cb"
Nov 03 11:29:47.640388 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:47.640328    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"
Nov 03 11:29:48.397598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:48.397391    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:49.181098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.178878    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:49.237075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.237045    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 11:29:49.282255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.282218    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-kube-api-access-bzlwf\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-kube-api-access-bzlwf\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.282455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.282286    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-token\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-token\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.282535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.282474    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-socket\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.282846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.282780    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-socket" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:49.282984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.282944    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-credential-socket\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.282984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.282975    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-data\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.283122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.283014    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-podinfo\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.283122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.283051    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-certs\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.283122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.283084    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istiod-ca-cert\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.283122 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.283115    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-envoy\") pod \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\" (UID: \"7cb2e513-e727-4117-90c0-0c33f7feb2c6\") "
Nov 03 11:29:49.283787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.283762    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.284879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.284852    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-data" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:49.285063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.284871    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-credential-socket" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:49.285161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.285059    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-certs" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:49.285359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.285335    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istiod-ca-cert" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:49.286473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.286448    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-envoy" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:49.286767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.286746    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-podinfo" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:29:49.289033 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.288969    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-kube-api-access-bzlwf" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "kube-api-access-bzlwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:49.290314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.290286    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-token" pod "7cb2e513-e727-4117-90c0-0c33f7feb2c6" (UID: "7cb2e513-e727-4117-90c0-0c33f7feb2c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:49.384924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.384894    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.384924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.384923    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.384924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.384933    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.384924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.384942    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.385205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.384954    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.385205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.384966    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.385205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.384980    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlwf\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-kube-api-access-bzlwf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.385205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.384993    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb2e513-e727-4117-90c0-0c33f7feb2c6-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.481034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.481002    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"]
Nov 03 11:29:49.488318 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.488288    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"]
Nov 03 11:29:49.501007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.500981    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"]
Nov 03 11:29:49.622051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.622014    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:49.658973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.658945    5817 generic.go:356] "Generic (PLEG): container finished" podID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerID="006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978" exitCode=0
Nov 03 11:29:49.659161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.659003    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"
Nov 03 11:29:49.659161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.659007    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" event={"ID":"7cb2e513-e727-4117-90c0-0c33f7feb2c6","Type":"ContainerDied","Data":"006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978"}
Nov 03 11:29:49.659287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.659173    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb" event={"ID":"7cb2e513-e727-4117-90c0-0c33f7feb2c6","Type":"ContainerDied","Data":"ccd5db0625a61f52ff42dafc6e283a295122ce8e8193cddafc3f3b25ec61b5b1"}
Nov 03 11:29:49.659287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.659214    5817 scope.go:122] "RemoveContainer" containerID="006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978"
Nov 03 11:29:49.671917 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.671893    5817 scope.go:122] "RemoveContainer" containerID="4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d"
Nov 03 11:29:49.682948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.682927    5817 scope.go:122] "RemoveContainer" containerID="006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978"
Nov 03 11:29:49.683232 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:49.683210    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978\": container with ID starting with 006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978 not found: ID does not exist" containerID="006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978"
Nov 03 11:29:49.683309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.683241    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978"} err="failed to get container status \"006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978\": rpc error: code = NotFound desc = could not find container \"006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978\": container with ID starting with 006a38088308b14c4818cbf7953158f79ed63d8a4619c7887e0f45bd53e01978 not found: ID does not exist"
Nov 03 11:29:49.683309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.683260    5817 scope.go:122] "RemoveContainer" containerID="4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d"
Nov 03 11:29:49.683495 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:49.683479    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d\": container with ID starting with 4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d not found: ID does not exist" containerID="4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d"
Nov 03 11:29:49.683562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.683500    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d"} err="failed to get container status \"4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d\": rpc error: code = NotFound desc = could not find container \"4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d\": container with ID starting with 4f211be525f736dbcaa13f066f8ebd0181c67b7220e3d6aecc6f25ee5bc5473d not found: ID does not exist"
Nov 03 11:29:49.684784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.684764    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"]
Nov 03 11:29:49.691246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.691225    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-fcc2ce-openshift-default-74968dd5f7-tlhdb"]
Nov 03 11:29:49.904040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.903961    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" path="/var/lib/kubelet/pods/7cb2e513-e727-4117-90c0-0c33f7feb2c6/volumes"
Nov 03 11:29:49.944115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:49.938708    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:50.555602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:50.552401    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:51.161720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:51.161489    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:51.812358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:51.804714    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:52.414831 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:52.414794    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:29:52.484596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:52.484356    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:53.164611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:53.164560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:53.886132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:53.886097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:54.115084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.115018    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 11:29:54.153307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.153275    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33600aaa-9f93-471c-951b-54362ed05f01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33600aaa-9f93-471c-951b-54362ed05f01-plugins-config-volume\") pod \"33600aaa-9f93-471c-951b-54362ed05f01\" (UID: \"33600aaa-9f93-471c-951b-54362ed05f01\") "
Nov 03 11:29:54.153632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.153603    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33600aaa-9f93-471c-951b-54362ed05f01-kube-api-access-xnlg8\" (UniqueName: \"kubernetes.io/projected/33600aaa-9f93-471c-951b-54362ed05f01-kube-api-access-xnlg8\") pod \"33600aaa-9f93-471c-951b-54362ed05f01\" (UID: \"33600aaa-9f93-471c-951b-54362ed05f01\") "
Nov 03 11:29:54.153801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.153666    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33600aaa-9f93-471c-951b-54362ed05f01-plugins-config-volume" pod "33600aaa-9f93-471c-951b-54362ed05f01" (UID: "33600aaa-9f93-471c-951b-54362ed05f01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:54.154238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.154217    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33600aaa-9f93-471c-951b-54362ed05f01-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:54.155923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.155898    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33600aaa-9f93-471c-951b-54362ed05f01-kube-api-access-xnlg8" pod "33600aaa-9f93-471c-951b-54362ed05f01" (UID: "33600aaa-9f93-471c-951b-54362ed05f01"). InnerVolumeSpecName "kube-api-access-xnlg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:54.255049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.255020    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlg8\" (UniqueName: \"kubernetes.io/projected/33600aaa-9f93-471c-951b-54362ed05f01-kube-api-access-xnlg8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:54.465602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.457969    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:54.682204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.682163    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"]
Nov 03 11:29:54.747928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.745947    5817 generic.go:356] "Generic (PLEG): container finished" podID="33600aaa-9f93-471c-951b-54362ed05f01" containerID="93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317" exitCode=0
Nov 03 11:29:54.747928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.746046    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"
Nov 03 11:29:54.747928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.746090    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw" event={"ID":"33600aaa-9f93-471c-951b-54362ed05f01","Type":"ContainerDied","Data":"93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317"}
Nov 03 11:29:54.747928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.746271    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw" event={"ID":"33600aaa-9f93-471c-951b-54362ed05f01","Type":"ContainerDied","Data":"02a7324464a9d4f96848dcd7381ecf531e7f3f966ba5d32c126f45720b00eada"}
Nov 03 11:29:54.747928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.746319    5817 scope.go:122] "RemoveContainer" containerID="93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317"
Nov 03 11:29:54.763658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.763641    5817 scope.go:122] "RemoveContainer" containerID="93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317"
Nov 03 11:29:54.763980 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:29:54.763950    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317\": container with ID starting with 93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317 not found: ID does not exist" containerID="93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317"
Nov 03 11:29:54.764084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.763990    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317"} err="failed to get container status \"93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317\": rpc error: code = NotFound desc = could not find container \"93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317\": container with ID starting with 93a554f35c087b3b27581d07b986a485f070cfe9c158bfd54fb256de78220317 not found: ID does not exist"
Nov 03 11:29:54.784048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.784014    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"]
Nov 03 11:29:54.790860 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.790837    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-fcc2ce-78ddfcdd5f-j9kfw"]
Nov 03 11:29:54.802520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.802484    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:54.941594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:54.941487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:55.096651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:55.091720    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:55.222893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:55.222851    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:55.815725 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:55.815685    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:55.905080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:55.905046    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33600aaa-9f93-471c-951b-54362ed05f01" path="/var/lib/kubelet/pods/33600aaa-9f93-471c-951b-54362ed05f01/volumes"
Nov 03 11:29:56.498329 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:56.497494    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:57.233592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.232937    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:57.671306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.671234    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"]
Nov 03 11:29:57.671889 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.671876    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy"
Nov 03 11:29:57.671948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.671891    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy"
Nov 03 11:29:57.671948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.671929    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f84c9456-bce4-4d4b-aea1-43680d02755f" containerName="revoke-keys"
Nov 03 11:29:57.671948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.671934    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84c9456-bce4-4d4b-aea1-43680d02755f" containerName="revoke-keys"
Nov 03 11:29:57.671948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.671946    5817 cpu_manager.go:414] "containerMap: removing container" podUID="33600aaa-9f93-471c-951b-54362ed05f01" containerName="payload-pre-processing"
Nov 03 11:29:57.672095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.671952    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33600aaa-9f93-471c-951b-54362ed05f01" containerName="payload-pre-processing"
Nov 03 11:29:57.672095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.672053    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="33600aaa-9f93-471c-951b-54362ed05f01" containerName="payload-pre-processing"
Nov 03 11:29:57.672095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.672064    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy"
Nov 03 11:29:57.672095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.672077    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84c9456-bce4-4d4b-aea1-43680d02755f" containerName="revoke-keys"
Nov 03 11:29:57.676868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.676840    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.682046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.682011    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"]
Nov 03 11:29:57.683380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.683340    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:29:57.683380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.683361    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:29:57.839353 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.839307    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.839562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.839411    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.839562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.839464    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jwt\" (UniqueName: \"kubernetes.io/projected/4c418ec8-5cf4-4185-8fe0-5291f681684a-kube-api-access-v2jwt\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.941059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.941021    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jwt\" (UniqueName: \"kubernetes.io/projected/4c418ec8-5cf4-4185-8fe0-5291f681684a-kube-api-access-v2jwt\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.941270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.941187    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.941270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.941235    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.941842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.941816    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.943831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.943807    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.949613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.949558    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jwt\" (UniqueName: \"kubernetes.io/projected/4c418ec8-5cf4-4185-8fe0-5291f681684a-kube-api-access-v2jwt\") pod \"maas-api-revoke-sub-451037cc-5psg7\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:57.988645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:57.980719    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:58.019239 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:58.019205    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:29:58.197826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:58.197741    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:29:58.314697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:58.314671    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"]
Nov 03 11:29:58.763325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:58.763227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:58.795083 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:58.795040    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7" event={"ID":"4c418ec8-5cf4-4185-8fe0-5291f681684a","Type":"ContainerStarted","Data":"c9885a5652d1dcc9172b55b7ea57ffb21b34ed28a190e7f957c15c95646841a1"}
Nov 03 11:29:58.795297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:58.795101    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7" event={"ID":"4c418ec8-5cf4-4185-8fe0-5291f681684a","Type":"ContainerStarted","Data":"92edad7cdbf28e656df7906a602a6bbb9a6b10d8e6a6d27f22772a724ef34fa7"}
Nov 03 11:29:58.818123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:58.818068    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7" podStartSLOduration=1.8180542800000001 podStartE2EDuration="1.81805428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:58.813074369 +0000 UTC m=+3629.018656420" watchObservedRunningTime="2026-11-03 11:29:58.81805428 +0000 UTC m=+3629.023636366"
Nov 03 11:29:59.668236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:59.668201    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:29:59.690125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:59.690080    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" podUID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" containerName="payload-processing" containerID="cri-o://135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32" gracePeriod=25
Nov 03 11:29:59.808684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:59.808566    5817 generic.go:356] "Generic (PLEG): container finished" podID="4c418ec8-5cf4-4185-8fe0-5291f681684a" containerID="c9885a5652d1dcc9172b55b7ea57ffb21b34ed28a190e7f957c15c95646841a1" exitCode=0
Nov 03 11:29:59.808842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:59.808675    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7" event={"ID":"4c418ec8-5cf4-4185-8fe0-5291f681684a","Type":"ContainerDied","Data":"c9885a5652d1dcc9172b55b7ea57ffb21b34ed28a190e7f957c15c95646841a1"}
Nov 03 11:29:59.846542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:29:59.846508    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" podUID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:30:00.142086 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.142002    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"]
Nov 03 11:30:00.143024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.142999    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy"
Nov 03 11:30:00.143024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.143023    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy"
Nov 03 11:30:00.143258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.143135    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb2e513-e727-4117-90c0-0c33f7feb2c6" containerName="istio-proxy"
Nov 03 11:30:00.146751 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.146734    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"]
Nov 03 11:30:00.146891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.146872    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:00.151362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.151342    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:00.151552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.151534    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"]
Nov 03 11:30:00.158463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.158431    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:00.161072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.161055    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"]
Nov 03 11:30:00.167068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.166794    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"]
Nov 03 11:30:00.171639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.171619    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"]
Nov 03 11:30:00.193050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.193029    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"]
Nov 03 11:30:00.199983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.199951    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"]
Nov 03 11:30:00.200119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.200081    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:00.205115 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.205096    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:00.215400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.215323    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"]
Nov 03 11:30:00.231091 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.231060    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"]
Nov 03 11:30:00.238999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.238974    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:00.246763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.246738    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"]
Nov 03 11:30:00.256774 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.253843    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:00.272855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.272828    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"]
Nov 03 11:30:00.279673 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.279649    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"]
Nov 03 11:30:00.284734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.284705    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mgp7\" (UniqueName: \"kubernetes.io/projected/6629ec48-f9c3-4fbe-9678-5725518af1f2-kube-api-access-5mgp7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh\" (UID: \"6629ec48-f9c3-4fbe-9678-5725518af1f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:00.284890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.284762    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rnmm\" (UniqueName: \"kubernetes.io/projected/8d6876d8-0a8e-4a9f-a750-6d617e44f545-kube-api-access-9rnmm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm\" (UID: \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:00.284890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.284818    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6629ec48-f9c3-4fbe-9678-5725518af1f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh\" (UID: \"6629ec48-f9c3-4fbe-9678-5725518af1f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:00.285032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.284974    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6876d8-0a8e-4a9f-a750-6d617e44f545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm\" (UID: \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:00.285092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.285045    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmggw\" (UniqueName: \"kubernetes.io/projected/775c5258-7f5d-4f58-ae04-f4449721275a-kube-api-access-hmggw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms\" (UID: \"775c5258-7f5d-4f58-ae04-f4449721275a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:00.285199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.285177    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9gc9\" (UniqueName: \"kubernetes.io/projected/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-kube-api-access-r9gc9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt\" (UID: \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:00.285322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.285217    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775c5258-7f5d-4f58-ae04-f4449721275a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms\" (UID: \"775c5258-7f5d-4f58-ae04-f4449721275a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:00.285322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.285264    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64212cd6-8eb3-451b-9a4a-d89c567772bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw\" (UID: \"64212cd6-8eb3-451b-9a4a-d89c567772bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:00.285322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.285301    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"]
Nov 03 11:30:00.285322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.285311    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt\" (UID: \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:00.285625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.285353    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdb9t\" (UniqueName: \"kubernetes.io/projected/64212cd6-8eb3-451b-9a4a-d89c567772bc-kube-api-access-sdb9t\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw\" (UID: \"64212cd6-8eb3-451b-9a4a-d89c567772bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:00.285625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.285404    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:00.294089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.294067    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"]
Nov 03 11:30:00.305331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.305311    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:00.309947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.309922    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"]
Nov 03 11:30:00.316374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.316352    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"]
Nov 03 11:30:00.324001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.323978    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"]
Nov 03 11:30:00.387189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387159    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmggw\" (UniqueName: \"kubernetes.io/projected/775c5258-7f5d-4f58-ae04-f4449721275a-kube-api-access-hmggw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms\" (UID: \"775c5258-7f5d-4f58-ae04-f4449721275a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:00.387348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387206    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrpx\" (UniqueName: \"kubernetes.io/projected/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-kube-api-access-zjrpx\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2\" (UID: \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:00.387348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387292    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9gc9\" (UniqueName: \"kubernetes.io/projected/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-kube-api-access-r9gc9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt\" (UID: \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:00.387733 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387705    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775c5258-7f5d-4f58-ae04-f4449721275a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms\" (UID: \"775c5258-7f5d-4f58-ae04-f4449721275a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:00.387821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387775    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64212cd6-8eb3-451b-9a4a-d89c567772bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw\" (UID: \"64212cd6-8eb3-451b-9a4a-d89c567772bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:00.387880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387820    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt\" (UID: \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:00.387880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387852    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2v2c\" (UniqueName: \"kubernetes.io/projected/e44d4c32-eab6-45d8-879d-134c5ac585c2-kube-api-access-j2v2c\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5\" (UID: \"e44d4c32-eab6-45d8-879d-134c5ac585c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:00.387996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387889    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdb9t\" (UniqueName: \"kubernetes.io/projected/64212cd6-8eb3-451b-9a4a-d89c567772bc-kube-api-access-sdb9t\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw\" (UID: \"64212cd6-8eb3-451b-9a4a-d89c567772bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:00.387996 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.387914    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75395ea0-7424-4cc5-a6a6-46150fc94050-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm\" (UID: \"75395ea0-7424-4cc5-a6a6-46150fc94050\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:00.388129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388004    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkxrc\" (UniqueName: \"kubernetes.io/projected/71b3de98-5049-4d08-a909-3d8311ec07ad-kube-api-access-gkxrc\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7\" (UID: \"71b3de98-5049-4d08-a909-3d8311ec07ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:00.388129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388059    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbtw\" (UniqueName: \"kubernetes.io/projected/75395ea0-7424-4cc5-a6a6-46150fc94050-kube-api-access-rfbtw\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm\" (UID: \"75395ea0-7424-4cc5-a6a6-46150fc94050\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:00.388129 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388093    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mgp7\" (UniqueName: \"kubernetes.io/projected/6629ec48-f9c3-4fbe-9678-5725518af1f2-kube-api-access-5mgp7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh\" (UID: \"6629ec48-f9c3-4fbe-9678-5725518af1f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:00.388303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388133    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnmm\" (UniqueName: \"kubernetes.io/projected/8d6876d8-0a8e-4a9f-a750-6d617e44f545-kube-api-access-9rnmm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm\" (UID: \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:00.388303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388156    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6629ec48-f9c3-4fbe-9678-5725518af1f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh\" (UID: \"6629ec48-f9c3-4fbe-9678-5725518af1f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:00.388303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388221    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2\" (UID: \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:00.388303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388259    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b3de98-5049-4d08-a909-3d8311ec07ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7\" (UID: \"71b3de98-5049-4d08-a909-3d8311ec07ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:00.388303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388301    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d4c32-eab6-45d8-879d-134c5ac585c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5\" (UID: \"e44d4c32-eab6-45d8-879d-134c5ac585c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:00.388614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388354    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6876d8-0a8e-4a9f-a750-6d617e44f545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm\" (UID: \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:00.388614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388547    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775c5258-7f5d-4f58-ae04-f4449721275a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms\" (UID: \"775c5258-7f5d-4f58-ae04-f4449721275a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:00.388614 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388591    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64212cd6-8eb3-451b-9a4a-d89c567772bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw\" (UID: \"64212cd6-8eb3-451b-9a4a-d89c567772bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:00.388796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.388612    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt\" (UID: \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:00.389186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.389162    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6876d8-0a8e-4a9f-a750-6d617e44f545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm\" (UID: \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:00.389259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.389219    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6629ec48-f9c3-4fbe-9678-5725518af1f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh\" (UID: \"6629ec48-f9c3-4fbe-9678-5725518af1f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:00.396445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.396314    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmggw\" (UniqueName: \"kubernetes.io/projected/775c5258-7f5d-4f58-ae04-f4449721275a-kube-api-access-hmggw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms\" (UID: \"775c5258-7f5d-4f58-ae04-f4449721275a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:00.397375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.397349    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9gc9\" (UniqueName: \"kubernetes.io/projected/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-kube-api-access-r9gc9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt\" (UID: \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:00.397491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.397353    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnmm\" (UniqueName: \"kubernetes.io/projected/8d6876d8-0a8e-4a9f-a750-6d617e44f545-kube-api-access-9rnmm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm\" (UID: \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:00.397491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.397460    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdb9t\" (UniqueName: \"kubernetes.io/projected/64212cd6-8eb3-451b-9a4a-d89c567772bc-kube-api-access-sdb9t\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw\" (UID: \"64212cd6-8eb3-451b-9a4a-d89c567772bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:00.397670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.397610    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mgp7\" (UniqueName: \"kubernetes.io/projected/6629ec48-f9c3-4fbe-9678-5725518af1f2-kube-api-access-5mgp7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh\" (UID: \"6629ec48-f9c3-4fbe-9678-5725518af1f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:00.468442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.468410    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:00.483762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.483740    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:00.489768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.489742    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrpx\" (UniqueName: \"kubernetes.io/projected/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-kube-api-access-zjrpx\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2\" (UID: \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:00.489904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.489890    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2v2c\" (UniqueName: \"kubernetes.io/projected/e44d4c32-eab6-45d8-879d-134c5ac585c2-kube-api-access-j2v2c\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5\" (UID: \"e44d4c32-eab6-45d8-879d-134c5ac585c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:00.489980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.489925    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75395ea0-7424-4cc5-a6a6-46150fc94050-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm\" (UID: \"75395ea0-7424-4cc5-a6a6-46150fc94050\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:00.489980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.489953    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkxrc\" (UniqueName: \"kubernetes.io/projected/71b3de98-5049-4d08-a909-3d8311ec07ad-kube-api-access-gkxrc\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7\" (UID: \"71b3de98-5049-4d08-a909-3d8311ec07ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:00.490109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.489987    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbtw\" (UniqueName: \"kubernetes.io/projected/75395ea0-7424-4cc5-a6a6-46150fc94050-kube-api-access-rfbtw\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm\" (UID: \"75395ea0-7424-4cc5-a6a6-46150fc94050\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:00.490109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.490055    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2\" (UID: \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:00.490109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.490090    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b3de98-5049-4d08-a909-3d8311ec07ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7\" (UID: \"71b3de98-5049-4d08-a909-3d8311ec07ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:00.490282 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.490125    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d4c32-eab6-45d8-879d-134c5ac585c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5\" (UID: \"e44d4c32-eab6-45d8-879d-134c5ac585c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:00.492003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.491031    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d4c32-eab6-45d8-879d-134c5ac585c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5\" (UID: \"e44d4c32-eab6-45d8-879d-134c5ac585c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:00.492003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.491033    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2\" (UID: \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:00.492003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.491555    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75395ea0-7424-4cc5-a6a6-46150fc94050-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm\" (UID: \"75395ea0-7424-4cc5-a6a6-46150fc94050\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:00.492003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.491713    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:00.492003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.491773    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b3de98-5049-4d08-a909-3d8311ec07ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7\" (UID: \"71b3de98-5049-4d08-a909-3d8311ec07ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:00.500020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.499808    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbtw\" (UniqueName: \"kubernetes.io/projected/75395ea0-7424-4cc5-a6a6-46150fc94050-kube-api-access-rfbtw\") pod \"maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm\" (UID: \"75395ea0-7424-4cc5-a6a6-46150fc94050\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:00.501098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.501054    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2v2c\" (UniqueName: \"kubernetes.io/projected/e44d4c32-eab6-45d8-879d-134c5ac585c2-kube-api-access-j2v2c\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5\" (UID: \"e44d4c32-eab6-45d8-879d-134c5ac585c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:00.502874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.502842    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkxrc\" (UniqueName: \"kubernetes.io/projected/71b3de98-5049-4d08-a909-3d8311ec07ad-kube-api-access-gkxrc\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7\" (UID: \"71b3de98-5049-4d08-a909-3d8311ec07ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:00.508152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.508107    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrpx\" (UniqueName: \"kubernetes.io/projected/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-kube-api-access-zjrpx\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2\" (UID: \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:00.536881 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.536847    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:00.569920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.558264    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:00.586255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.576212    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:00.599958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.599927    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:00.623926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.623822    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:00.634638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.631817    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:00.680659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.670459    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:00.980291 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:00.980264    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64212cd6_8eb3_451b_9a4a_d89c567772bc.slice/crio-1949cbf9092628ab1d6aa2046bedde3428b64d7ab8fad5b18a2726326cef88f3 WatchSource:0}: Error finding container 1949cbf9092628ab1d6aa2046bedde3428b64d7ab8fad5b18a2726326cef88f3: Status 404 returned error can't find the container with id 1949cbf9092628ab1d6aa2046bedde3428b64d7ab8fad5b18a2726326cef88f3
Nov 03 11:30:00.981090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:00.981060    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"]
Nov 03 11:30:01.001362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.001336    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"]
Nov 03 11:30:01.003103 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:01.003076    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775c5258_7f5d_4f58_ae04_f4449721275a.slice/crio-ba54c2de33c2d464055fec574829ce0d448abc8e89081427c3f142d6e8f97032 WatchSource:0}: Error finding container ba54c2de33c2d464055fec574829ce0d448abc8e89081427c3f142d6e8f97032: Status 404 returned error can't find the container with id ba54c2de33c2d464055fec574829ce0d448abc8e89081427c3f142d6e8f97032
Nov 03 11:30:01.053918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.053893    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"]
Nov 03 11:30:01.113362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.113333    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:30:01.200103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.200074    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"]
Nov 03 11:30:01.205536 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.205507    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"]
Nov 03 11:30:01.209241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.208548    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-xks-ca\") pod \"4c418ec8-5cf4-4185-8fe0-5291f681684a\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") "
Nov 03 11:30:01.209241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.208789    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c418ec8-5cf4-4185-8fe0-5291f681684a-kube-api-access-v2jwt\" (UniqueName: \"kubernetes.io/projected/4c418ec8-5cf4-4185-8fe0-5291f681684a-kube-api-access-v2jwt\") pod \"4c418ec8-5cf4-4185-8fe0-5291f681684a\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") "
Nov 03 11:30:01.209241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.208824    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-openshift-ca\") pod \"4c418ec8-5cf4-4185-8fe0-5291f681684a\" (UID: \"4c418ec8-5cf4-4185-8fe0-5291f681684a\") "
Nov 03 11:30:01.209544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.209519    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-openshift-ca" pod "4c418ec8-5cf4-4185-8fe0-5291f681684a" (UID: "4c418ec8-5cf4-4185-8fe0-5291f681684a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:01.212080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.212051    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c418ec8-5cf4-4185-8fe0-5291f681684a-kube-api-access-v2jwt" pod "4c418ec8-5cf4-4185-8fe0-5291f681684a" (UID: "4c418ec8-5cf4-4185-8fe0-5291f681684a"). InnerVolumeSpecName "kube-api-access-v2jwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:01.213456 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:01.213035    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b3de98_5049_4d08_a909_3d8311ec07ad.slice/crio-39150d55df765962890ceb58c45375abff1893caefd6e337f8658406e2bff8a4 WatchSource:0}: Error finding container 39150d55df765962890ceb58c45375abff1893caefd6e337f8658406e2bff8a4: Status 404 returned error can't find the container with id 39150d55df765962890ceb58c45375abff1893caefd6e337f8658406e2bff8a4
Nov 03 11:30:01.213456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.213043    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-xks-ca" pod "4c418ec8-5cf4-4185-8fe0-5291f681684a" (UID: "4c418ec8-5cf4-4185-8fe0-5291f681684a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:01.213456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.213121    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"]
Nov 03 11:30:01.218716 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.218692    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"]
Nov 03 11:30:01.266314 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:01.266283    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44d4c32_eab6_45d8_879d_134c5ac585c2.slice/crio-f1026007ad311b8d58da06a84c3793bc6ff3c376c42edc951cb13fcbbeb76a82 WatchSource:0}: Error finding container f1026007ad311b8d58da06a84c3793bc6ff3c376c42edc951cb13fcbbeb76a82: Status 404 returned error can't find the container with id f1026007ad311b8d58da06a84c3793bc6ff3c376c42edc951cb13fcbbeb76a82
Nov 03 11:30:01.268387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.268305    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"]
Nov 03 11:30:01.279413 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.279385    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"]
Nov 03 11:30:01.310179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.309973    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jwt\" (UniqueName: \"kubernetes.io/projected/4c418ec8-5cf4-4185-8fe0-5291f681684a-kube-api-access-v2jwt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:01.310179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.310004    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:01.310179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.310021    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c418ec8-5cf4-4185-8fe0-5291f681684a-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:01.485662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.483897    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:01.833938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.833908    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" event={"ID":"6629ec48-f9c3-4fbe-9678-5725518af1f2","Type":"ContainerStarted","Data":"9d0ce14f3dae52491c9cbcca9cf56a24ffeda375fb66d3c903137f31a0fe148b"}
Nov 03 11:30:01.834427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.833975    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" event={"ID":"6629ec48-f9c3-4fbe-9678-5725518af1f2","Type":"ContainerStarted","Data":"db969074c7461923b685c5c3e6d1818ad3681aac2e4381929e410d6555d507ad"}
Nov 03 11:30:01.835562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.835532    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" event={"ID":"8d6876d8-0a8e-4a9f-a750-6d617e44f545","Type":"ContainerStarted","Data":"2d672f19b05c1a7dce5537621eefbbdf9657a6a171af52163f6f6b429f174b08"}
Nov 03 11:30:01.837552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.837521    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" event={"ID":"775c5258-7f5d-4f58-ae04-f4449721275a","Type":"ContainerStarted","Data":"df628388b86ddc2bf9cdde64f2b3f8b80b299521af678938fcdf5ddf079e9ac7"}
Nov 03 11:30:01.837808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.837591    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" event={"ID":"775c5258-7f5d-4f58-ae04-f4449721275a","Type":"ContainerStarted","Data":"ba54c2de33c2d464055fec574829ce0d448abc8e89081427c3f142d6e8f97032"}
Nov 03 11:30:01.839565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.839540    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" event={"ID":"5d50f1d2-4dca-4f89-beef-be956cbfdb2f","Type":"ContainerStarted","Data":"c9d45efd496c8744d53524b290417139b9cc953b42870a3c0d05fb4df0cc87da"}
Nov 03 11:30:01.839701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.839604    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" event={"ID":"5d50f1d2-4dca-4f89-beef-be956cbfdb2f","Type":"ContainerStarted","Data":"9451fbce726b488bd862a4cb0634d3fb5989b03a7855c3ccaf8ecf6992281c28"}
Nov 03 11:30:01.841013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.840988    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" event={"ID":"71b3de98-5049-4d08-a909-3d8311ec07ad","Type":"ContainerStarted","Data":"39150d55df765962890ceb58c45375abff1893caefd6e337f8658406e2bff8a4"}
Nov 03 11:30:01.842481 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.842460    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" event={"ID":"b6b0fd1d-92bb-4031-846c-4e10c24c2b22","Type":"ContainerStarted","Data":"87dae77454706e9b8e7d3a6e1d86714d1fbc041d3f207d3b98a823b8b21e64a3"}
Nov 03 11:30:01.844455 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.844414    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" event={"ID":"64212cd6-8eb3-451b-9a4a-d89c567772bc","Type":"ContainerStarted","Data":"5913c204bba2db80ccae14ab3e6dfbf1ec473a6ce12fbca86b7cf841a8bd534b"}
Nov 03 11:30:01.844613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.844455    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" event={"ID":"64212cd6-8eb3-451b-9a4a-d89c567772bc","Type":"ContainerStarted","Data":"1949cbf9092628ab1d6aa2046bedde3428b64d7ab8fad5b18a2726326cef88f3"}
Nov 03 11:30:01.846169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.846096    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" event={"ID":"e44d4c32-eab6-45d8-879d-134c5ac585c2","Type":"ContainerStarted","Data":"f1026007ad311b8d58da06a84c3793bc6ff3c376c42edc951cb13fcbbeb76a82"}
Nov 03 11:30:01.847643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.847623    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"
Nov 03 11:30:01.847790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.847651    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7" event={"ID":"4c418ec8-5cf4-4185-8fe0-5291f681684a","Type":"ContainerDied","Data":"92edad7cdbf28e656df7906a602a6bbb9a6b10d8e6a6d27f22772a724ef34fa7"}
Nov 03 11:30:01.847790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.847693    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92edad7cdbf28e656df7906a602a6bbb9a6b10d8e6a6d27f22772a724ef34fa7"
Nov 03 11:30:01.855439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.855409    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" event={"ID":"75395ea0-7424-4cc5-a6a6-46150fc94050","Type":"ContainerStarted","Data":"f5f307a53c640510425e3c8f43126d4008789061e59a52e40f797a9f5957ecf0"}
Nov 03 11:30:01.855599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.855466    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" event={"ID":"75395ea0-7424-4cc5-a6a6-46150fc94050","Type":"ContainerStarted","Data":"fc6528803130fceb3357e65797b33dc9b12b6690e6c51e12c7cac2cf23fd33d0"}
Nov 03 11:30:01.860521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.860481    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" podStartSLOduration=1.860468954 podStartE2EDuration="1.860468954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:01.852749844 +0000 UTC m=+3632.058331900" watchObservedRunningTime="2026-11-03 11:30:01.860468954 +0000 UTC m=+3632.066051006"
Nov 03 11:30:01.871690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.871653    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" podStartSLOduration=1.8716421909999998 podStartE2EDuration="1.871642191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:01.870352985 +0000 UTC m=+3632.075935036" watchObservedRunningTime="2026-11-03 11:30:01.871642191 +0000 UTC m=+3632.077224242"
Nov 03 11:30:01.892306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.891844    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" podStartSLOduration=1.891827529 podStartE2EDuration="1.891827529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:01.884605066 +0000 UTC m=+3632.090187110" watchObservedRunningTime="2026-11-03 11:30:01.891827529 +0000 UTC m=+3632.097409583"
Nov 03 11:30:01.904274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.904227    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" podStartSLOduration=1.9042143440000001 podStartE2EDuration="1.904214344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:01.899997992 +0000 UTC m=+3632.105580062" watchObservedRunningTime="2026-11-03 11:30:01.904214344 +0000 UTC m=+3632.109796430"
Nov 03 11:30:01.924859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:01.924790    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" podStartSLOduration=1.924747761 podStartE2EDuration="1.924747761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:01.918796662 +0000 UTC m=+3632.124378720" watchObservedRunningTime="2026-11-03 11:30:01.924747761 +0000 UTC m=+3632.130329819"
Nov 03 11:30:02.144648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.141768    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:02.795808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.795639    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:02.868791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.868758    5817 generic.go:356] "Generic (PLEG): container finished" podID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerID="5441d8a38beaba6295be036f9daefbf4a27f6769a274ebf240180aaf94dce8c7" exitCode=60
Nov 03 11:30:02.869216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.868839    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" event={"ID":"b6b0fd1d-92bb-4031-846c-4e10c24c2b22","Type":"ContainerDied","Data":"5441d8a38beaba6295be036f9daefbf4a27f6769a274ebf240180aaf94dce8c7"}
Nov 03 11:30:02.869216 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.869172    5817 scope.go:122] "RemoveContainer" containerID="5441d8a38beaba6295be036f9daefbf4a27f6769a274ebf240180aaf94dce8c7"
Nov 03 11:30:02.870767 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.870744    5817 generic.go:356] "Generic (PLEG): container finished" podID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerID="5913c204bba2db80ccae14ab3e6dfbf1ec473a6ce12fbca86b7cf841a8bd534b" exitCode=60
Nov 03 11:30:02.870896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.870824    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" event={"ID":"64212cd6-8eb3-451b-9a4a-d89c567772bc","Type":"ContainerDied","Data":"5913c204bba2db80ccae14ab3e6dfbf1ec473a6ce12fbca86b7cf841a8bd534b"}
Nov 03 11:30:02.871063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.871044    5817 scope.go:122] "RemoveContainer" containerID="5913c204bba2db80ccae14ab3e6dfbf1ec473a6ce12fbca86b7cf841a8bd534b"
Nov 03 11:30:02.877484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.877460    5817 generic.go:356] "Generic (PLEG): container finished" podID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerID="a24a463d2062b6028a57c04c7ebe598d1c609416ea316750477e1f030c966b88" exitCode=60
Nov 03 11:30:02.877564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.877500    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" event={"ID":"e44d4c32-eab6-45d8-879d-134c5ac585c2","Type":"ContainerDied","Data":"a24a463d2062b6028a57c04c7ebe598d1c609416ea316750477e1f030c966b88"}
Nov 03 11:30:02.878159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.877873    5817 scope.go:122] "RemoveContainer" containerID="a24a463d2062b6028a57c04c7ebe598d1c609416ea316750477e1f030c966b88"
Nov 03 11:30:02.879522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.879501    5817 generic.go:356] "Generic (PLEG): container finished" podID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerID="f5f307a53c640510425e3c8f43126d4008789061e59a52e40f797a9f5957ecf0" exitCode=60
Nov 03 11:30:02.879650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.879587    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" event={"ID":"75395ea0-7424-4cc5-a6a6-46150fc94050","Type":"ContainerDied","Data":"f5f307a53c640510425e3c8f43126d4008789061e59a52e40f797a9f5957ecf0"}
Nov 03 11:30:02.879814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.879795    5817 scope.go:122] "RemoveContainer" containerID="f5f307a53c640510425e3c8f43126d4008789061e59a52e40f797a9f5957ecf0"
Nov 03 11:30:02.881450 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.881426    5817 generic.go:356] "Generic (PLEG): container finished" podID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerID="9d0ce14f3dae52491c9cbcca9cf56a24ffeda375fb66d3c903137f31a0fe148b" exitCode=60
Nov 03 11:30:02.881686 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.881514    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" event={"ID":"6629ec48-f9c3-4fbe-9678-5725518af1f2","Type":"ContainerDied","Data":"9d0ce14f3dae52491c9cbcca9cf56a24ffeda375fb66d3c903137f31a0fe148b"}
Nov 03 11:30:02.881783 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.881754    5817 scope.go:122] "RemoveContainer" containerID="9d0ce14f3dae52491c9cbcca9cf56a24ffeda375fb66d3c903137f31a0fe148b"
Nov 03 11:30:02.883513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.883492    5817 generic.go:356] "Generic (PLEG): container finished" podID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerID="153e55dddb6905ab36f44feaf1ac333aad27139952d7ec7146ba373f88113e5d" exitCode=60
Nov 03 11:30:02.883632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.883621    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" event={"ID":"8d6876d8-0a8e-4a9f-a750-6d617e44f545","Type":"ContainerDied","Data":"153e55dddb6905ab36f44feaf1ac333aad27139952d7ec7146ba373f88113e5d"}
Nov 03 11:30:02.883906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.883889    5817 scope.go:122] "RemoveContainer" containerID="153e55dddb6905ab36f44feaf1ac333aad27139952d7ec7146ba373f88113e5d"
Nov 03 11:30:02.895794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.893091    5817 generic.go:356] "Generic (PLEG): container finished" podID="775c5258-7f5d-4f58-ae04-f4449721275a" containerID="df628388b86ddc2bf9cdde64f2b3f8b80b299521af678938fcdf5ddf079e9ac7" exitCode=60
Nov 03 11:30:02.895794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.893197    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" event={"ID":"775c5258-7f5d-4f58-ae04-f4449721275a","Type":"ContainerDied","Data":"df628388b86ddc2bf9cdde64f2b3f8b80b299521af678938fcdf5ddf079e9ac7"}
Nov 03 11:30:02.895794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.893523    5817 scope.go:122] "RemoveContainer" containerID="df628388b86ddc2bf9cdde64f2b3f8b80b299521af678938fcdf5ddf079e9ac7"
Nov 03 11:30:02.898121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.898092    5817 generic.go:356] "Generic (PLEG): container finished" podID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerID="c9d45efd496c8744d53524b290417139b9cc953b42870a3c0d05fb4df0cc87da" exitCode=60
Nov 03 11:30:02.898263 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.898134    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" event={"ID":"5d50f1d2-4dca-4f89-beef-be956cbfdb2f","Type":"ContainerDied","Data":"c9d45efd496c8744d53524b290417139b9cc953b42870a3c0d05fb4df0cc87da"}
Nov 03 11:30:02.898666 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.898645    5817 scope.go:122] "RemoveContainer" containerID="c9d45efd496c8744d53524b290417139b9cc953b42870a3c0d05fb4df0cc87da"
Nov 03 11:30:02.904715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.904692    5817 generic.go:356] "Generic (PLEG): container finished" podID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerID="5bf21157c3f880a25a1efbb3b6c98a9733745347a6829fc2141a2bcd065e8f11" exitCode=60
Nov 03 11:30:02.904817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.904734    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" event={"ID":"71b3de98-5049-4d08-a909-3d8311ec07ad","Type":"ContainerDied","Data":"5bf21157c3f880a25a1efbb3b6c98a9733745347a6829fc2141a2bcd065e8f11"}
Nov 03 11:30:02.904966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:02.904949    5817 scope.go:122] "RemoveContainer" containerID="5bf21157c3f880a25a1efbb3b6c98a9733745347a6829fc2141a2bcd065e8f11"
Nov 03 11:30:03.590594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.587522    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:03.915192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.915160    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" event={"ID":"71b3de98-5049-4d08-a909-3d8311ec07ad","Type":"ContainerStarted","Data":"3a9a17c823bb6ca3899c297cee6111c3159174b28030ec5acb0e734fd02ee1fc"}
Nov 03 11:30:03.917305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.917282    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" event={"ID":"b6b0fd1d-92bb-4031-846c-4e10c24c2b22","Type":"ContainerStarted","Data":"190eabc07c9df44da2c03f3508f37772c7dcd9f32c7442e656a3c496f24d6cd3"}
Nov 03 11:30:03.919547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.919523    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" event={"ID":"64212cd6-8eb3-451b-9a4a-d89c567772bc","Type":"ContainerStarted","Data":"cf475bc0c77ed8ac798d97850b5353b448dcfdd2b92fae8f58b073ca33ab61d8"}
Nov 03 11:30:03.921618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.921585    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" event={"ID":"e44d4c32-eab6-45d8-879d-134c5ac585c2","Type":"ContainerStarted","Data":"06aa5ac658e78b73f7ed1366b3893967e8b8ba83647a6d46e78adf5479dcd33d"}
Nov 03 11:30:03.923793 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.923771    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" event={"ID":"75395ea0-7424-4cc5-a6a6-46150fc94050","Type":"ContainerStarted","Data":"916bc3b439603fe64f51167bdb7a451cbe60ac009dd50b8838fd057c5120b7c7"}
Nov 03 11:30:03.925942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.925918    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" event={"ID":"6629ec48-f9c3-4fbe-9678-5725518af1f2","Type":"ContainerStarted","Data":"b9d13da8f8c5dc8857b930ef0134c84a9d8afe13444a10d87e7622efb142a333"}
Nov 03 11:30:03.928243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.928221    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" event={"ID":"8d6876d8-0a8e-4a9f-a750-6d617e44f545","Type":"ContainerStarted","Data":"273191bea60af1ae8efadbe6de935671b81837980b6537e872d9f9bb4d35d5dd"}
Nov 03 11:30:03.930558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.930219    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:03.932433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.932407    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" event={"ID":"775c5258-7f5d-4f58-ae04-f4449721275a","Type":"ContainerStarted","Data":"fc1e319152280fb854b7c9361ad599209d66c443a440edf471f4c47e07bcc091"}
Nov 03 11:30:03.997691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:03.967380    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" podStartSLOduration=3.967358635 podStartE2EDuration="3.967358635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:03.930906102 +0000 UTC m=+3634.136488153" watchObservedRunningTime="2026-11-03 11:30:03.967358635 +0000 UTC m=+3634.172940687"
Nov 03 11:30:04.009861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.007944    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" podStartSLOduration=4.00790451 podStartE2EDuration="4.00790451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:03.999931403 +0000 UTC m=+3634.205513455" watchObservedRunningTime="2026-11-03 11:30:04.00790451 +0000 UTC m=+3634.213486561"
Nov 03 11:30:04.044839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.011238    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" event={"ID":"5d50f1d2-4dca-4f89-beef-be956cbfdb2f","Type":"ContainerStarted","Data":"32204510861ea6b9b91ff805c95fe0ab2d365bdf1a7a1d19d7d1f4adc9157b6b"}
Nov 03 11:30:04.103400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.103334    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" podStartSLOduration=4.103314822 podStartE2EDuration="4.103314822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:04.093963523 +0000 UTC m=+3634.299545583" watchObservedRunningTime="2026-11-03 11:30:04.103314822 +0000 UTC m=+3634.308896886"
Nov 03 11:30:04.117398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.117332    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" podStartSLOduration=4.117309928 podStartE2EDuration="4.117309928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:04.112733016 +0000 UTC m=+3634.318315075" watchObservedRunningTime="2026-11-03 11:30:04.117309928 +0000 UTC m=+3634.322891987"
Nov 03 11:30:04.370021 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:04.369994    5817 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d50f1d2_4dca_4f89_beef_be956cbfdb2f.slice/crio-conmon-32204510861ea6b9b91ff805c95fe0ab2d365bdf1a7a1d19d7d1f4adc9157b6b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:30:04.503944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.503895    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:04.511713 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.511683    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main" containerID="cri-o://13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799" gracePeriod=45
Nov 03 11:30:04.517753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.517724    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main" containerID="cri-o://8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c" gracePeriod=45
Nov 03 11:30:04.530150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.530118    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" containerID="cri-o://d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12" gracePeriod=45
Nov 03 11:30:04.551826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.551797    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused"
Nov 03 11:30:04.945781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.945744    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 11:30:04.949644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.949623    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 11:30:04.957999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.957975    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 11:30:04.977957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.977926    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-model-cache\") pod \"10f80020-b416-45e1-b616-ee44e0668e45\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") "
Nov 03 11:30:04.978132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.977986    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tls-certs\") pod \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") "
Nov 03 11:30:04.978132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978023    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-dshm\") pod \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") "
Nov 03 11:30:04.978132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978055    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f80020-b416-45e1-b616-ee44e0668e45-kube-api-access-vccx5\" (UniqueName: \"kubernetes.io/projected/10f80020-b416-45e1-b616-ee44e0668e45-kube-api-access-vccx5\") pod \"10f80020-b416-45e1-b616-ee44e0668e45\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") "
Nov 03 11:30:04.978132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978089    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-dshm\") pod \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") "
Nov 03 11:30:04.978389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978145    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-home\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-home\") pod \"10f80020-b416-45e1-b616-ee44e0668e45\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") "
Nov 03 11:30:04.978389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978179    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc83089-3cd2-4b50-94c7-fc527102cbaa-kube-api-access-h96dr\" (UniqueName: \"kubernetes.io/projected/4bc83089-3cd2-4b50-94c7-fc527102cbaa-kube-api-access-h96dr\") pod \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") "
Nov 03 11:30:04.978389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978213    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tmp-dir\") pod \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") "
Nov 03 11:30:04.978389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978219    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-model-cache" pod "10f80020-b416-45e1-b616-ee44e0668e45" (UID: "10f80020-b416-45e1-b616-ee44e0668e45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.978389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978246    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-model-cache\") pod \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") "
Nov 03 11:30:04.978389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978281    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-home\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-home\") pod \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") "
Nov 03 11:30:04.978389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978338    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-home\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-home\") pod \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") "
Nov 03 11:30:04.978389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978370    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f80020-b416-45e1-b616-ee44e0668e45-tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80020-b416-45e1-b616-ee44e0668e45-tls-certs\") pod \"10f80020-b416-45e1-b616-ee44e0668e45\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") "
Nov 03 11:30:04.979045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978412    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe82e65-7284-4584-9ebe-65b1b6338eaa-kube-api-access-jwldd\" (UniqueName: \"kubernetes.io/projected/4fe82e65-7284-4584-9ebe-65b1b6338eaa-kube-api-access-jwldd\") pod \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") "
Nov 03 11:30:04.979045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978439    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-dshm\") pod \"10f80020-b416-45e1-b616-ee44e0668e45\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") "
Nov 03 11:30:04.979045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978487    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-model-cache\") pod \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\" (UID: \"4bc83089-3cd2-4b50-94c7-fc527102cbaa\") "
Nov 03 11:30:04.979045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978526    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-tmp-dir\") pod \"10f80020-b416-45e1-b616-ee44e0668e45\" (UID: \"10f80020-b416-45e1-b616-ee44e0668e45\") "
Nov 03 11:30:04.979045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978563    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tls-certs\") pod \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") "
Nov 03 11:30:04.979045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.978697    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tmp-dir\") pod \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\" (UID: \"4fe82e65-7284-4584-9ebe-65b1b6338eaa\") "
Nov 03 11:30:04.979449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979046    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-home" pod "4fe82e65-7284-4584-9ebe-65b1b6338eaa" (UID: "4fe82e65-7284-4584-9ebe-65b1b6338eaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.979449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979244    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-model-cache" pod "4fe82e65-7284-4584-9ebe-65b1b6338eaa" (UID: "4fe82e65-7284-4584-9ebe-65b1b6338eaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.979648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979621    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tmp-dir" pod "4bc83089-3cd2-4b50-94c7-fc527102cbaa" (UID: "4bc83089-3cd2-4b50-94c7-fc527102cbaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.979754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979656    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:04.979754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979679    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:04.979754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979694    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:04.979754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979699    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-model-cache" pod "4bc83089-3cd2-4b50-94c7-fc527102cbaa" (UID: "4bc83089-3cd2-4b50-94c7-fc527102cbaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.980076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979823    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-home" pod "10f80020-b416-45e1-b616-ee44e0668e45" (UID: "10f80020-b416-45e1-b616-ee44e0668e45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.980076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.979885    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-tmp-dir" pod "10f80020-b416-45e1-b616-ee44e0668e45" (UID: "10f80020-b416-45e1-b616-ee44e0668e45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.980866 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.980588    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-home" pod "4bc83089-3cd2-4b50-94c7-fc527102cbaa" (UID: "4bc83089-3cd2-4b50-94c7-fc527102cbaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.981646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.981611    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tmp-dir" pod "4fe82e65-7284-4584-9ebe-65b1b6338eaa" (UID: "4fe82e65-7284-4584-9ebe-65b1b6338eaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.983719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.983675    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe82e65-7284-4584-9ebe-65b1b6338eaa-kube-api-access-jwldd" pod "4fe82e65-7284-4584-9ebe-65b1b6338eaa" (UID: "4fe82e65-7284-4584-9ebe-65b1b6338eaa"). InnerVolumeSpecName "kube-api-access-jwldd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:04.990909 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.990865    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f80020-b416-45e1-b616-ee44e0668e45-tls-certs" pod "10f80020-b416-45e1-b616-ee44e0668e45" (UID: "10f80020-b416-45e1-b616-ee44e0668e45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:04.999001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.998955    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-dshm" pod "10f80020-b416-45e1-b616-ee44e0668e45" (UID: "10f80020-b416-45e1-b616-ee44e0668e45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.999001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.998970    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tls-certs" pod "4fe82e65-7284-4584-9ebe-65b1b6338eaa" (UID: "4fe82e65-7284-4584-9ebe-65b1b6338eaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:04.999187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.999134    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc83089-3cd2-4b50-94c7-fc527102cbaa-kube-api-access-h96dr" pod "4bc83089-3cd2-4b50-94c7-fc527102cbaa" (UID: "4bc83089-3cd2-4b50-94c7-fc527102cbaa"). InnerVolumeSpecName "kube-api-access-h96dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:04.999496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.999348    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f80020-b416-45e1-b616-ee44e0668e45-kube-api-access-vccx5" pod "10f80020-b416-45e1-b616-ee44e0668e45" (UID: "10f80020-b416-45e1-b616-ee44e0668e45"). InnerVolumeSpecName "kube-api-access-vccx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:04.999496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.999414    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-dshm" pod "4bc83089-3cd2-4b50-94c7-fc527102cbaa" (UID: "4bc83089-3cd2-4b50-94c7-fc527102cbaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.999496 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:04.999465    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-dshm" pod "4fe82e65-7284-4584-9ebe-65b1b6338eaa" (UID: "4fe82e65-7284-4584-9ebe-65b1b6338eaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:05.003487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.003450    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tls-certs" pod "4bc83089-3cd2-4b50-94c7-fc527102cbaa" (UID: "4bc83089-3cd2-4b50-94c7-fc527102cbaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:05.028281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.028249    5817 generic.go:356] "Generic (PLEG): container finished" podID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerID="b9d13da8f8c5dc8857b930ef0134c84a9d8afe13444a10d87e7622efb142a333" exitCode=60
Nov 03 11:30:05.028456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.028361    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" event={"ID":"6629ec48-f9c3-4fbe-9678-5725518af1f2","Type":"ContainerDied","Data":"b9d13da8f8c5dc8857b930ef0134c84a9d8afe13444a10d87e7622efb142a333"}
Nov 03 11:30:05.028456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.028417    5817 scope.go:122] "RemoveContainer" containerID="9d0ce14f3dae52491c9cbcca9cf56a24ffeda375fb66d3c903137f31a0fe148b"
Nov 03 11:30:05.032227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.031168    5817 generic.go:356] "Generic (PLEG): container finished" podID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerID="273191bea60af1ae8efadbe6de935671b81837980b6537e872d9f9bb4d35d5dd" exitCode=60
Nov 03 11:30:05.032227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.031201    5817 scope.go:122] "RemoveContainer" containerID="b9d13da8f8c5dc8857b930ef0134c84a9d8afe13444a10d87e7622efb142a333"
Nov 03 11:30:05.032227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.031238    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" event={"ID":"8d6876d8-0a8e-4a9f-a750-6d617e44f545","Type":"ContainerDied","Data":"273191bea60af1ae8efadbe6de935671b81837980b6537e872d9f9bb4d35d5dd"}
Nov 03 11:30:05.032227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.031610    5817 scope.go:122] "RemoveContainer" containerID="273191bea60af1ae8efadbe6de935671b81837980b6537e872d9f9bb4d35d5dd"
Nov 03 11:30:05.032227 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.031625    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh_odh-ai-gateway-infra(6629ec48-f9c3-4fbe-9678-5725518af1f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2"
Nov 03 11:30:05.032227 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.031940    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm_odh-ai-gateway-infra(8d6876d8-0a8e-4a9f-a750-6d617e44f545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545"
Nov 03 11:30:05.039236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.035210    5817 generic.go:356] "Generic (PLEG): container finished" podID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerID="d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12" exitCode=0
Nov 03 11:30:05.039236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.035306    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" event={"ID":"4fe82e65-7284-4584-9ebe-65b1b6338eaa","Type":"ContainerDied","Data":"d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12"}
Nov 03 11:30:05.039236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.035335    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng" event={"ID":"4fe82e65-7284-4584-9ebe-65b1b6338eaa","Type":"ContainerDied","Data":"5ac394a14fe8b21dbaa4d94c376b878fafe8e33e7501c744bb4b4d0cd104a3e8"}
Nov 03 11:30:05.039236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.035404    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"
Nov 03 11:30:05.048080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.042655    5817 generic.go:356] "Generic (PLEG): container finished" podID="775c5258-7f5d-4f58-ae04-f4449721275a" containerID="fc1e319152280fb854b7c9361ad599209d66c443a440edf471f4c47e07bcc091" exitCode=60
Nov 03 11:30:05.048080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.042766    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" event={"ID":"775c5258-7f5d-4f58-ae04-f4449721275a","Type":"ContainerDied","Data":"fc1e319152280fb854b7c9361ad599209d66c443a440edf471f4c47e07bcc091"}
Nov 03 11:30:05.048080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.043132    5817 scope.go:122] "RemoveContainer" containerID="fc1e319152280fb854b7c9361ad599209d66c443a440edf471f4c47e07bcc091"
Nov 03 11:30:05.048080 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.043457    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms_odh-ai-gateway-infra(775c5258-7f5d-4f58-ae04-f4449721275a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" podUID="775c5258-7f5d-4f58-ae04-f4449721275a"
Nov 03 11:30:05.050683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.049732    5817 generic.go:356] "Generic (PLEG): container finished" podID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerID="32204510861ea6b9b91ff805c95fe0ab2d365bdf1a7a1d19d7d1f4adc9157b6b" exitCode=60
Nov 03 11:30:05.050683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.049844    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" event={"ID":"5d50f1d2-4dca-4f89-beef-be956cbfdb2f","Type":"ContainerDied","Data":"32204510861ea6b9b91ff805c95fe0ab2d365bdf1a7a1d19d7d1f4adc9157b6b"}
Nov 03 11:30:05.050683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.050143    5817 scope.go:122] "RemoveContainer" containerID="32204510861ea6b9b91ff805c95fe0ab2d365bdf1a7a1d19d7d1f4adc9157b6b"
Nov 03 11:30:05.051038 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.051017    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt_odh-ai-gateway-infra(5d50f1d2-4dca-4f89-beef-be956cbfdb2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f"
Nov 03 11:30:05.051853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.051831    5817 generic.go:356] "Generic (PLEG): container finished" podID="10f80020-b416-45e1-b616-ee44e0668e45" containerID="8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c" exitCode=0
Nov 03 11:30:05.051981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.051876    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"
Nov 03 11:30:05.051981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.051928    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" event={"ID":"10f80020-b416-45e1-b616-ee44e0668e45","Type":"ContainerDied","Data":"8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c"}
Nov 03 11:30:05.051981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.051952    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp" event={"ID":"10f80020-b416-45e1-b616-ee44e0668e45","Type":"ContainerDied","Data":"04a07cf3391143c5479b02ed269137df39bd6341701d1e5d25fc52a5e2522a25"}
Nov 03 11:30:05.054471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.054444    5817 generic.go:356] "Generic (PLEG): container finished" podID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerID="3a9a17c823bb6ca3899c297cee6111c3159174b28030ec5acb0e734fd02ee1fc" exitCode=60
Nov 03 11:30:05.054636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.054544    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" event={"ID":"71b3de98-5049-4d08-a909-3d8311ec07ad","Type":"ContainerDied","Data":"3a9a17c823bb6ca3899c297cee6111c3159174b28030ec5acb0e734fd02ee1fc"}
Nov 03 11:30:05.054949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.054929    5817 scope.go:122] "RemoveContainer" containerID="3a9a17c823bb6ca3899c297cee6111c3159174b28030ec5acb0e734fd02ee1fc"
Nov 03 11:30:05.055232 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.055207    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7_odh-ai-gateway-infra(71b3de98-5049-4d08-a909-3d8311ec07ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad"
Nov 03 11:30:05.055939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.055662    5817 scope.go:122] "RemoveContainer" containerID="153e55dddb6905ab36f44feaf1ac333aad27139952d7ec7146ba373f88113e5d"
Nov 03 11:30:05.060350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.057409    5817 generic.go:356] "Generic (PLEG): container finished" podID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerID="190eabc07c9df44da2c03f3508f37772c7dcd9f32c7442e656a3c496f24d6cd3" exitCode=60
Nov 03 11:30:05.060350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.057477    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" event={"ID":"b6b0fd1d-92bb-4031-846c-4e10c24c2b22","Type":"ContainerDied","Data":"190eabc07c9df44da2c03f3508f37772c7dcd9f32c7442e656a3c496f24d6cd3"}
Nov 03 11:30:05.060350 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.057843    5817 scope.go:122] "RemoveContainer" containerID="190eabc07c9df44da2c03f3508f37772c7dcd9f32c7442e656a3c496f24d6cd3"
Nov 03 11:30:05.060350 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.058120    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2_odh-ai-gateway-infra(b6b0fd1d-92bb-4031-846c-4e10c24c2b22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22"
Nov 03 11:30:05.064765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.061179    5817 generic.go:356] "Generic (PLEG): container finished" podID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerID="cf475bc0c77ed8ac798d97850b5353b448dcfdd2b92fae8f58b073ca33ab61d8" exitCode=60
Nov 03 11:30:05.064765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.061259    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" event={"ID":"64212cd6-8eb3-451b-9a4a-d89c567772bc","Type":"ContainerDied","Data":"cf475bc0c77ed8ac798d97850b5353b448dcfdd2b92fae8f58b073ca33ab61d8"}
Nov 03 11:30:05.064765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.061590    5817 scope.go:122] "RemoveContainer" containerID="cf475bc0c77ed8ac798d97850b5353b448dcfdd2b92fae8f58b073ca33ab61d8"
Nov 03 11:30:05.064765 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.061834    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw_odh-ai-gateway-infra(64212cd6-8eb3-451b-9a4a-d89c567772bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc"
Nov 03 11:30:05.068623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.067690    5817 generic.go:356] "Generic (PLEG): container finished" podID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerID="06aa5ac658e78b73f7ed1366b3893967e8b8ba83647a6d46e78adf5479dcd33d" exitCode=60
Nov 03 11:30:05.068623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.067758    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" event={"ID":"e44d4c32-eab6-45d8-879d-134c5ac585c2","Type":"ContainerDied","Data":"06aa5ac658e78b73f7ed1366b3893967e8b8ba83647a6d46e78adf5479dcd33d"}
Nov 03 11:30:05.068623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.068073    5817 scope.go:122] "RemoveContainer" containerID="06aa5ac658e78b73f7ed1366b3893967e8b8ba83647a6d46e78adf5479dcd33d"
Nov 03 11:30:05.068623 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.068247    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5_odh-ai-gateway-infra(e44d4c32-eab6-45d8-879d-134c5ac585c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2"
Nov 03 11:30:05.070910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.070889    5817 generic.go:356] "Generic (PLEG): container finished" podID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerID="13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799" exitCode=0
Nov 03 11:30:05.071038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.070999    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" event={"ID":"4bc83089-3cd2-4b50-94c7-fc527102cbaa","Type":"ContainerDied","Data":"13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799"}
Nov 03 11:30:05.071038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.071021    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst" event={"ID":"4bc83089-3cd2-4b50-94c7-fc527102cbaa","Type":"ContainerDied","Data":"8039d15120499e98f782a0814e7a696152e1747b7bbe151177793b6fe59a8036"}
Nov 03 11:30:05.071233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.071075    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"
Nov 03 11:30:05.078360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.078330    5817 scope.go:122] "RemoveContainer" containerID="d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12"
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.080241    5817 generic.go:356] "Generic (PLEG): container finished" podID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerID="916bc3b439603fe64f51167bdb7a451cbe60ac009dd50b8838fd057c5120b7c7" exitCode=60
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.080358    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" event={"ID":"75395ea0-7424-4cc5-a6a6-46150fc94050","Type":"ContainerDied","Data":"916bc3b439603fe64f51167bdb7a451cbe60ac009dd50b8838fd057c5120b7c7"}
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.080678    5817 scope.go:122] "RemoveContainer" containerID="916bc3b439603fe64f51167bdb7a451cbe60ac009dd50b8838fd057c5120b7c7"
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.080912    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm_odh-ai-gateway-infra(75395ea0-7424-4cc5-a6a6-46150fc94050)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050"
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081002    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081031    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081046    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081059    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081073    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081087    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e65-7284-4584-9ebe-65b1b6338eaa-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081101    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vccx5\" (UniqueName: \"kubernetes.io/projected/10f80020-b416-45e1-b616-ee44e0668e45-kube-api-access-vccx5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081115    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081129    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081142    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h96dr\" (UniqueName: \"kubernetes.io/projected/4bc83089-3cd2-4b50-94c7-fc527102cbaa-kube-api-access-h96dr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081156    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081170    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc83089-3cd2-4b50-94c7-fc527102cbaa-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081185    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80020-b416-45e1-b616-ee44e0668e45-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081199    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwldd\" (UniqueName: \"kubernetes.io/projected/4fe82e65-7284-4584-9ebe-65b1b6338eaa-kube-api-access-jwldd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.081779 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.081215    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80020-b416-45e1-b616-ee44e0668e45-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.096615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.096591    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"]
Nov 03 11:30:05.097917 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.097897    5817 scope.go:122] "RemoveContainer" containerID="d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12"
Nov 03 11:30:05.098279 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.098256    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12\": container with ID starting with d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12 not found: ID does not exist" containerID="d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12"
Nov 03 11:30:05.098348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.098291    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12"} err="failed to get container status \"d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12\": rpc error: code = NotFound desc = could not find container \"d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12\": container with ID starting with d82c7ca68083638bcf22e7b5b1ff11f82f93076aedc7cf1badf0f98f5a981e12 not found: ID does not exist"
Nov 03 11:30:05.098348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.098314    5817 scope.go:122] "RemoveContainer" containerID="df628388b86ddc2bf9cdde64f2b3f8b80b299521af678938fcdf5ddf079e9ac7"
Nov 03 11:30:05.100984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.100960    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/simulator-w4-d5d3c3-kserve-865c8fd5ff-gk2ng"]
Nov 03 11:30:05.113827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.113811    5817 scope.go:122] "RemoveContainer" containerID="c9d45efd496c8744d53524b290417139b9cc953b42870a3c0d05fb4df0cc87da"
Nov 03 11:30:05.129683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.129660    5817 scope.go:122] "RemoveContainer" containerID="8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c"
Nov 03 11:30:05.145841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.145818    5817 scope.go:122] "RemoveContainer" containerID="8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c"
Nov 03 11:30:05.146170 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.146148    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c\": container with ID starting with 8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c not found: ID does not exist" containerID="8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c"
Nov 03 11:30:05.146236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.146180    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c"} err="failed to get container status \"8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c\": rpc error: code = NotFound desc = could not find container \"8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c\": container with ID starting with 8542e45b9f285e6e6120f6e64891148925ca409515a185bda8ef2df2bc0bcf3c not found: ID does not exist"
Nov 03 11:30:05.146236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.146197    5817 scope.go:122] "RemoveContainer" containerID="5bf21157c3f880a25a1efbb3b6c98a9733745347a6829fc2141a2bcd065e8f11"
Nov 03 11:30:05.160361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.160338    5817 scope.go:122] "RemoveContainer" containerID="5441d8a38beaba6295be036f9daefbf4a27f6769a274ebf240180aaf94dce8c7"
Nov 03 11:30:05.174755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.174737    5817 scope.go:122] "RemoveContainer" containerID="5913c204bba2db80ccae14ab3e6dfbf1ec473a6ce12fbca86b7cf841a8bd534b"
Nov 03 11:30:05.198087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.198062    5817 scope.go:122] "RemoveContainer" containerID="a24a463d2062b6028a57c04c7ebe598d1c609416ea316750477e1f030c966b88"
Nov 03 11:30:05.221295 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.221272    5817 scope.go:122] "RemoveContainer" containerID="13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799"
Nov 03 11:30:05.233186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.233168    5817 scope.go:122] "RemoveContainer" containerID="13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799"
Nov 03 11:30:05.233493 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:05.233464    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799\": container with ID starting with 13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799 not found: ID does not exist" containerID="13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799"
Nov 03 11:30:05.233613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.233498    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799"} err="failed to get container status \"13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799\": rpc error: code = NotFound desc = could not find container \"13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799\": container with ID starting with 13775e86ad33a3c1dee37b263dece461ca187275583bf55427d05c7ea6c21799 not found: ID does not exist"
Nov 03 11:30:05.233613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.233519    5817 scope.go:122] "RemoveContainer" containerID="f5f307a53c640510425e3c8f43126d4008789061e59a52e40f797a9f5957ecf0"
Nov 03 11:30:05.294667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.294637    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"]
Nov 03 11:30:05.321297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.321246    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/premium-w4-d5d3c3-kserve-55cbb58654-j6hkp"]
Nov 03 11:30:05.419313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.418947    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"]
Nov 03 11:30:05.434171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.434081    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:05.464187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.464153    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-w4-d5d3c3-kserve-7787f5d4f7-zdzst"]
Nov 03 11:30:05.475759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.475726    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"]
Nov 03 11:30:05.476898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.476872    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4c418ec8-5cf4-4185-8fe0-5291f681684a" containerName="revoke-keys"
Nov 03 11:30:05.476898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.476899    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c418ec8-5cf4-4185-8fe0-5291f681684a" containerName="revoke-keys"
Nov 03 11:30:05.477114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.476932    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main"
Nov 03 11:30:05.477114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.476940    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main"
Nov 03 11:30:05.477114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.476960    5817 cpu_manager.go:414] "containerMap: removing container" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main"
Nov 03 11:30:05.477114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.476967    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main"
Nov 03 11:30:05.477114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.476977    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main"
Nov 03 11:30:05.477114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.476984    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main"
Nov 03 11:30:05.477360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.477146    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c418ec8-5cf4-4185-8fe0-5291f681684a" containerName="revoke-keys"
Nov 03 11:30:05.477360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.477160    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" containerName="main"
Nov 03 11:30:05.477360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.477169    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f80020-b416-45e1-b616-ee44e0668e45" containerName="main"
Nov 03 11:30:05.477360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.477191    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" containerName="main"
Nov 03 11:30:05.483721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.483697    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"]
Nov 03 11:30:05.483923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.483827    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.489749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.489721    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.489875 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.489844    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjrr\" (UniqueName: \"kubernetes.io/projected/19875dc4-8c13-4271-95c6-c03df5861569-kube-api-access-2kjrr\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.489951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.489889    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.491634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.491611    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:30:05.589495 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.589456    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:05.591098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.591062    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.591256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.591197    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjrr\" (UniqueName: \"kubernetes.io/projected/19875dc4-8c13-4271-95c6-c03df5861569-kube-api-access-2kjrr\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.591256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.591241    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.592823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.592798    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.594643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.594619    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.603913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.603884    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjrr\" (UniqueName: \"kubernetes.io/projected/19875dc4-8c13-4271-95c6-c03df5861569-kube-api-access-2kjrr\") pod \"maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.806485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.806455    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:05.914773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.914739    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f80020-b416-45e1-b616-ee44e0668e45" path="/var/lib/kubelet/pods/10f80020-b416-45e1-b616-ee44e0668e45/volumes"
Nov 03 11:30:05.915302 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.915279    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc83089-3cd2-4b50-94c7-fc527102cbaa" path="/var/lib/kubelet/pods/4bc83089-3cd2-4b50-94c7-fc527102cbaa/volumes"
Nov 03 11:30:05.915787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:05.915769    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe82e65-7284-4584-9ebe-65b1b6338eaa" path="/var/lib/kubelet/pods/4fe82e65-7284-4584-9ebe-65b1b6338eaa/volumes"
Nov 03 11:30:06.087898 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:06.087866    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19875dc4_8c13_4271_95c6_c03df5861569.slice/crio-e6f840e92ec58c1aa1ee60094b31ffa9d47109a9ee5d0797f991a8b8f978352b WatchSource:0}: Error finding container e6f840e92ec58c1aa1ee60094b31ffa9d47109a9ee5d0797f991a8b8f978352b: Status 404 returned error can't find the container with id e6f840e92ec58c1aa1ee60094b31ffa9d47109a9ee5d0797f991a8b8f978352b
Nov 03 11:30:06.089053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.089026    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"]
Nov 03 11:30:06.093852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.093830    5817 scope.go:122] "RemoveContainer" containerID="b9d13da8f8c5dc8857b930ef0134c84a9d8afe13444a10d87e7622efb142a333"
Nov 03 11:30:06.094192 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.094165    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh_odh-ai-gateway-infra(6629ec48-f9c3-4fbe-9678-5725518af1f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2"
Nov 03 11:30:06.095630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.095609    5817 scope.go:122] "RemoveContainer" containerID="273191bea60af1ae8efadbe6de935671b81837980b6537e872d9f9bb4d35d5dd"
Nov 03 11:30:06.095931 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.095913    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm_odh-ai-gateway-infra(8d6876d8-0a8e-4a9f-a750-6d617e44f545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545"
Nov 03 11:30:06.098289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.098267    5817 scope.go:122] "RemoveContainer" containerID="fc1e319152280fb854b7c9361ad599209d66c443a440edf471f4c47e07bcc091"
Nov 03 11:30:06.098530 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.098508    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms_odh-ai-gateway-infra(775c5258-7f5d-4f58-ae04-f4449721275a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" podUID="775c5258-7f5d-4f58-ae04-f4449721275a"
Nov 03 11:30:06.107837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.107808    5817 scope.go:122] "RemoveContainer" containerID="32204510861ea6b9b91ff805c95fe0ab2d365bdf1a7a1d19d7d1f4adc9157b6b"
Nov 03 11:30:06.109615 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.108087    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt_odh-ai-gateway-infra(5d50f1d2-4dca-4f89-beef-be956cbfdb2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f"
Nov 03 11:30:06.111299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.111281    5817 scope.go:122] "RemoveContainer" containerID="3a9a17c823bb6ca3899c297cee6111c3159174b28030ec5acb0e734fd02ee1fc"
Nov 03 11:30:06.111714 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.111689    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7_odh-ai-gateway-infra(71b3de98-5049-4d08-a909-3d8311ec07ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad"
Nov 03 11:30:06.113421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.113399    5817 scope.go:122] "RemoveContainer" containerID="190eabc07c9df44da2c03f3508f37772c7dcd9f32c7442e656a3c496f24d6cd3"
Nov 03 11:30:06.113690 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.113666    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2_odh-ai-gateway-infra(b6b0fd1d-92bb-4031-846c-4e10c24c2b22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22"
Nov 03 11:30:06.115232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.115209    5817 scope.go:122] "RemoveContainer" containerID="cf475bc0c77ed8ac798d97850b5353b448dcfdd2b92fae8f58b073ca33ab61d8"
Nov 03 11:30:06.115439 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.115420    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw_odh-ai-gateway-infra(64212cd6-8eb3-451b-9a4a-d89c567772bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc"
Nov 03 11:30:06.117057 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.117041    5817 scope.go:122] "RemoveContainer" containerID="06aa5ac658e78b73f7ed1366b3893967e8b8ba83647a6d46e78adf5479dcd33d"
Nov 03 11:30:06.117267 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.117248    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5_odh-ai-gateway-infra(e44d4c32-eab6-45d8-879d-134c5ac585c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2"
Nov 03 11:30:06.120120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.120101    5817 scope.go:122] "RemoveContainer" containerID="916bc3b439603fe64f51167bdb7a451cbe60ac009dd50b8838fd057c5120b7c7"
Nov 03 11:30:06.120349 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:06.120328    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm_odh-ai-gateway-infra(75395ea0-7424-4cc5-a6a6-46150fc94050)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050"
Nov 03 11:30:06.583501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:06.579405    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:07.131475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:07.131440    5817 generic.go:356] "Generic (PLEG): container finished" podID="19875dc4-8c13-4271-95c6-c03df5861569" containerID="36f2b13faee3a7c10ffedc0b3d8389f714d2a58af67b4a5746c847cef758d3f9" exitCode=0
Nov 03 11:30:07.132007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:07.131528    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7" event={"ID":"19875dc4-8c13-4271-95c6-c03df5861569","Type":"ContainerDied","Data":"36f2b13faee3a7c10ffedc0b3d8389f714d2a58af67b4a5746c847cef758d3f9"}
Nov 03 11:30:07.132007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:07.131592    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7" event={"ID":"19875dc4-8c13-4271-95c6-c03df5861569","Type":"ContainerStarted","Data":"e6f840e92ec58c1aa1ee60094b31ffa9d47109a9ee5d0797f991a8b8f978352b"}
Nov 03 11:30:07.455691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:07.455657    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:08.328062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.328032    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:08.332123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.332101    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19875dc4-8c13-4271-95c6-c03df5861569-kube-api-access-2kjrr\" (UniqueName: \"kubernetes.io/projected/19875dc4-8c13-4271-95c6-c03df5861569-kube-api-access-2kjrr\") pod \"19875dc4-8c13-4271-95c6-c03df5861569\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") "
Nov 03 11:30:08.332237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.332146    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-openshift-ca\") pod \"19875dc4-8c13-4271-95c6-c03df5861569\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") "
Nov 03 11:30:08.332290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.332228    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-xks-ca\") pod \"19875dc4-8c13-4271-95c6-c03df5861569\" (UID: \"19875dc4-8c13-4271-95c6-c03df5861569\") "
Nov 03 11:30:08.332526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.332502    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-openshift-ca" pod "19875dc4-8c13-4271-95c6-c03df5861569" (UID: "19875dc4-8c13-4271-95c6-c03df5861569"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:08.333005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.332982    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:08.334948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.334925    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-xks-ca" pod "19875dc4-8c13-4271-95c6-c03df5861569" (UID: "19875dc4-8c13-4271-95c6-c03df5861569"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:08.335035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.334957    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19875dc4-8c13-4271-95c6-c03df5861569-kube-api-access-2kjrr" pod "19875dc4-8c13-4271-95c6-c03df5861569" (UID: "19875dc4-8c13-4271-95c6-c03df5861569"). InnerVolumeSpecName "kube-api-access-2kjrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:08.434613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.434560    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kjrr\" (UniqueName: \"kubernetes.io/projected/19875dc4-8c13-4271-95c6-c03df5861569-kube-api-access-2kjrr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:08.434613 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.434610    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19875dc4-8c13-4271-95c6-c03df5861569-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:08.520604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.518043    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:08.797345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:08.797251    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:09.151894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:09.151818    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"
Nov 03 11:30:09.151894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:09.151860    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7" event={"ID":"19875dc4-8c13-4271-95c6-c03df5861569","Type":"ContainerDied","Data":"e6f840e92ec58c1aa1ee60094b31ffa9d47109a9ee5d0797f991a8b8f978352b"}
Nov 03 11:30:09.152096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:09.151913    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6f840e92ec58c1aa1ee60094b31ffa9d47109a9ee5d0797f991a8b8f978352b"
Nov 03 11:30:09.382624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:09.373090    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:09.846351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:09.846308    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" podUID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:30:10.082611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:10.081142    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:10.827192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:10.827157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:11.452604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:11.451627    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:12.059596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:12.058824    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:12.698870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:12.698833    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:13.329482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:13.328813    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:14.041117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:14.041076    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:14.681692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:14.681656    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:14.894270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:14.894238    5817 scope.go:122] "RemoveContainer" containerID="916bc3b439603fe64f51167bdb7a451cbe60ac009dd50b8838fd057c5120b7c7"
Nov 03 11:30:14.894448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:14.894398    5817 scope.go:122] "RemoveContainer" containerID="cf475bc0c77ed8ac798d97850b5353b448dcfdd2b92fae8f58b073ca33ab61d8"
Nov 03 11:30:14.894532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:14.894501    5817 scope.go:122] "RemoveContainer" containerID="fc1e319152280fb854b7c9361ad599209d66c443a440edf471f4c47e07bcc091"
Nov 03 11:30:15.466965 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:15.466933    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:15.617511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:15.609469    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"]
Nov 03 11:30:15.775598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:15.775292    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:15.897954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:15.897869    5817 scope.go:122] "RemoveContainer" containerID="190eabc07c9df44da2c03f3508f37772c7dcd9f32c7442e656a3c496f24d6cd3"
Nov 03 11:30:15.898126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:15.898104    5817 scope.go:122] "RemoveContainer" containerID="273191bea60af1ae8efadbe6de935671b81837980b6537e872d9f9bb4d35d5dd"
Nov 03 11:30:15.898883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:15.898857    5817 scope.go:122] "RemoveContainer" containerID="32204510861ea6b9b91ff805c95fe0ab2d365bdf1a7a1d19d7d1f4adc9157b6b"
Nov 03 11:30:15.899026 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:15.899004    5817 scope.go:122] "RemoveContainer" containerID="b9d13da8f8c5dc8857b930ef0134c84a9d8afe13444a10d87e7622efb142a333"
Nov 03 11:30:16.046070 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.044143    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:16.214712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.214602    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:16.222910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.221505    5817 generic.go:356] "Generic (PLEG): container finished" podID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerID="3470510a5a41a87dc50206e912eb28489523f9fd2570aaabdc7f138de5e7a9ce" exitCode=60
Nov 03 11:30:16.222910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.221657    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" event={"ID":"64212cd6-8eb3-451b-9a4a-d89c567772bc","Type":"ContainerDied","Data":"3470510a5a41a87dc50206e912eb28489523f9fd2570aaabdc7f138de5e7a9ce"}
Nov 03 11:30:16.222910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.221717    5817 scope.go:122] "RemoveContainer" containerID="cf475bc0c77ed8ac798d97850b5353b448dcfdd2b92fae8f58b073ca33ab61d8"
Nov 03 11:30:16.222910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.222176    5817 scope.go:122] "RemoveContainer" containerID="3470510a5a41a87dc50206e912eb28489523f9fd2570aaabdc7f138de5e7a9ce"
Nov 03 11:30:16.222910 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:16.222478    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw_odh-ai-gateway-infra(64212cd6-8eb3-451b-9a4a-d89c567772bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc"
Nov 03 11:30:16.227107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.227054    5817 generic.go:356] "Generic (PLEG): container finished" podID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerID="1e92a4f6ade28b4199995f58b527594671d2594bea5d7e1d8a0f43d6f409496c" exitCode=60
Nov 03 11:30:16.227348 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.227325    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" event={"ID":"75395ea0-7424-4cc5-a6a6-46150fc94050","Type":"ContainerDied","Data":"1e92a4f6ade28b4199995f58b527594671d2594bea5d7e1d8a0f43d6f409496c"}
Nov 03 11:30:16.227755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.227739    5817 scope.go:122] "RemoveContainer" containerID="1e92a4f6ade28b4199995f58b527594671d2594bea5d7e1d8a0f43d6f409496c"
Nov 03 11:30:16.228148 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:16.228125    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm_odh-ai-gateway-infra(75395ea0-7424-4cc5-a6a6-46150fc94050)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050"
Nov 03 11:30:16.233303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.233280    5817 generic.go:356] "Generic (PLEG): container finished" podID="775c5258-7f5d-4f58-ae04-f4449721275a" containerID="600a7061c8e3af9bad731ec46b21e3263cfc69f78a0ffb0ddc0e0f08ec93abc4" exitCode=60
Nov 03 11:30:16.233437 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.233420    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" event={"ID":"775c5258-7f5d-4f58-ae04-f4449721275a","Type":"ContainerDied","Data":"600a7061c8e3af9bad731ec46b21e3263cfc69f78a0ffb0ddc0e0f08ec93abc4"}
Nov 03 11:30:16.233833 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.233808    5817 scope.go:122] "RemoveContainer" containerID="600a7061c8e3af9bad731ec46b21e3263cfc69f78a0ffb0ddc0e0f08ec93abc4"
Nov 03 11:30:16.234263 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:16.234157    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms_odh-ai-gateway-infra(775c5258-7f5d-4f58-ae04-f4449721275a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" podUID="775c5258-7f5d-4f58-ae04-f4449721275a"
Nov 03 11:30:16.248832 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.248514    5817 scope.go:122] "RemoveContainer" containerID="916bc3b439603fe64f51167bdb7a451cbe60ac009dd50b8838fd057c5120b7c7"
Nov 03 11:30:16.264396 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.264378    5817 scope.go:122] "RemoveContainer" containerID="fc1e319152280fb854b7c9361ad599209d66c443a440edf471f4c47e07bcc091"
Nov 03 11:30:16.359138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:16.359106    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:17.081886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.081846    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:17.245139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.245101    5817 generic.go:356] "Generic (PLEG): container finished" podID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerID="3d099895bf0dc94677baf1e926e6225161d0d02424c3f7f9436ca18070859efc" exitCode=60
Nov 03 11:30:17.245268 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.245139    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" event={"ID":"6629ec48-f9c3-4fbe-9678-5725518af1f2","Type":"ContainerDied","Data":"3d099895bf0dc94677baf1e926e6225161d0d02424c3f7f9436ca18070859efc"}
Nov 03 11:30:17.245268 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.245229    5817 scope.go:122] "RemoveContainer" containerID="b9d13da8f8c5dc8857b930ef0134c84a9d8afe13444a10d87e7622efb142a333"
Nov 03 11:30:17.245669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.245654    5817 scope.go:122] "RemoveContainer" containerID="3d099895bf0dc94677baf1e926e6225161d0d02424c3f7f9436ca18070859efc"
Nov 03 11:30:17.245968 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:17.245949    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh_odh-ai-gateway-infra(6629ec48-f9c3-4fbe-9678-5725518af1f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2"
Nov 03 11:30:17.247764 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.247732    5817 generic.go:356] "Generic (PLEG): container finished" podID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerID="807395b582e452af4829bf9e30813e39729c7c468b44bb49f0b4d5be58a9b320" exitCode=60
Nov 03 11:30:17.247872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.247768    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" event={"ID":"8d6876d8-0a8e-4a9f-a750-6d617e44f545","Type":"ContainerDied","Data":"807395b582e452af4829bf9e30813e39729c7c468b44bb49f0b4d5be58a9b320"}
Nov 03 11:30:17.248121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.248108    5817 scope.go:122] "RemoveContainer" containerID="807395b582e452af4829bf9e30813e39729c7c468b44bb49f0b4d5be58a9b320"
Nov 03 11:30:17.248508 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:17.248488    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm_odh-ai-gateway-infra(8d6876d8-0a8e-4a9f-a750-6d617e44f545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545"
Nov 03 11:30:17.253167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.253145    5817 generic.go:356] "Generic (PLEG): container finished" podID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerID="c77843cd5a1492db61626f01bc3873d162a47b4f9165b451a8af0a61531968ff" exitCode=60
Nov 03 11:30:17.253291 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.253224    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" event={"ID":"5d50f1d2-4dca-4f89-beef-be956cbfdb2f","Type":"ContainerDied","Data":"c77843cd5a1492db61626f01bc3873d162a47b4f9165b451a8af0a61531968ff"}
Nov 03 11:30:17.253637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.253618    5817 scope.go:122] "RemoveContainer" containerID="c77843cd5a1492db61626f01bc3873d162a47b4f9165b451a8af0a61531968ff"
Nov 03 11:30:17.253910 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:17.253883    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt_odh-ai-gateway-infra(5d50f1d2-4dca-4f89-beef-be956cbfdb2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f"
Nov 03 11:30:17.255728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.255705    5817 generic.go:356] "Generic (PLEG): container finished" podID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerID="04f4f708fbe82f6857a9e2515948aa4ce35194be815ae3ed30068dd228ef8440" exitCode=60
Nov 03 11:30:17.255837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.255777    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" event={"ID":"b6b0fd1d-92bb-4031-846c-4e10c24c2b22","Type":"ContainerDied","Data":"04f4f708fbe82f6857a9e2515948aa4ce35194be815ae3ed30068dd228ef8440"}
Nov 03 11:30:17.256150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.256135    5817 scope.go:122] "RemoveContainer" containerID="04f4f708fbe82f6857a9e2515948aa4ce35194be815ae3ed30068dd228ef8440"
Nov 03 11:30:17.256401 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:17.256381    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2_odh-ai-gateway-infra(b6b0fd1d-92bb-4031-846c-4e10c24c2b22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22"
Nov 03 11:30:17.256940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.256918    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"]
Nov 03 11:30:17.277056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.275739    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"]
Nov 03 11:30:17.288000 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.287960    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"]
Nov 03 11:30:17.381787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.381759    5817 scope.go:122] "RemoveContainer" containerID="273191bea60af1ae8efadbe6de935671b81837980b6537e872d9f9bb4d35d5dd"
Nov 03 11:30:17.403174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.403152    5817 scope.go:122] "RemoveContainer" containerID="32204510861ea6b9b91ff805c95fe0ab2d365bdf1a7a1d19d7d1f4adc9157b6b"
Nov 03 11:30:17.432510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.432484    5817 scope.go:122] "RemoveContainer" containerID="190eabc07c9df44da2c03f3508f37772c7dcd9f32c7442e656a3c496f24d6cd3"
Nov 03 11:30:17.525092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.525061    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:17.570622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.565721    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:17.588645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.587157    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:17.626056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.626022    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64212cd6-8eb3-451b-9a4a-d89c567772bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64212cd6-8eb3-451b-9a4a-d89c567772bc-maas-api-service-ca\") pod \"64212cd6-8eb3-451b-9a4a-d89c567772bc\" (UID: \"64212cd6-8eb3-451b-9a4a-d89c567772bc\") "
Nov 03 11:30:17.626434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.626410    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64212cd6-8eb3-451b-9a4a-d89c567772bc-kube-api-access-sdb9t\" (UniqueName: \"kubernetes.io/projected/64212cd6-8eb3-451b-9a4a-d89c567772bc-kube-api-access-sdb9t\") pod \"64212cd6-8eb3-451b-9a4a-d89c567772bc\" (UID: \"64212cd6-8eb3-451b-9a4a-d89c567772bc\") "
Nov 03 11:30:17.628002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.627936    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64212cd6-8eb3-451b-9a4a-d89c567772bc-maas-api-service-ca" pod "64212cd6-8eb3-451b-9a4a-d89c567772bc" (UID: "64212cd6-8eb3-451b-9a4a-d89c567772bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:17.630058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.630031    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64212cd6-8eb3-451b-9a4a-d89c567772bc-kube-api-access-sdb9t" pod "64212cd6-8eb3-451b-9a4a-d89c567772bc" (UID: "64212cd6-8eb3-451b-9a4a-d89c567772bc"). InnerVolumeSpecName "kube-api-access-sdb9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:17.727915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.727881    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75395ea0-7424-4cc5-a6a6-46150fc94050-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75395ea0-7424-4cc5-a6a6-46150fc94050-maas-api-service-ca\") pod \"75395ea0-7424-4cc5-a6a6-46150fc94050\" (UID: \"75395ea0-7424-4cc5-a6a6-46150fc94050\") "
Nov 03 11:30:17.728107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.727948    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75395ea0-7424-4cc5-a6a6-46150fc94050-kube-api-access-rfbtw\" (UniqueName: \"kubernetes.io/projected/75395ea0-7424-4cc5-a6a6-46150fc94050-kube-api-access-rfbtw\") pod \"75395ea0-7424-4cc5-a6a6-46150fc94050\" (UID: \"75395ea0-7424-4cc5-a6a6-46150fc94050\") "
Nov 03 11:30:17.728171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.728114    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775c5258-7f5d-4f58-ae04-f4449721275a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775c5258-7f5d-4f58-ae04-f4449721275a-maas-api-service-ca\") pod \"775c5258-7f5d-4f58-ae04-f4449721275a\" (UID: \"775c5258-7f5d-4f58-ae04-f4449721275a\") "
Nov 03 11:30:17.728249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.728231    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775c5258-7f5d-4f58-ae04-f4449721275a-kube-api-access-hmggw\" (UniqueName: \"kubernetes.io/projected/775c5258-7f5d-4f58-ae04-f4449721275a-kube-api-access-hmggw\") pod \"775c5258-7f5d-4f58-ae04-f4449721275a\" (UID: \"775c5258-7f5d-4f58-ae04-f4449721275a\") "
Nov 03 11:30:17.728310 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.728286    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75395ea0-7424-4cc5-a6a6-46150fc94050-maas-api-service-ca" pod "75395ea0-7424-4cc5-a6a6-46150fc94050" (UID: "75395ea0-7424-4cc5-a6a6-46150fc94050"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:17.728732 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.728705    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775c5258-7f5d-4f58-ae04-f4449721275a-maas-api-service-ca" pod "775c5258-7f5d-4f58-ae04-f4449721275a" (UID: "775c5258-7f5d-4f58-ae04-f4449721275a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:17.728846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.728800    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64212cd6-8eb3-451b-9a4a-d89c567772bc-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:17.728846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.728818    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75395ea0-7424-4cc5-a6a6-46150fc94050-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:17.728846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.728833    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdb9t\" (UniqueName: \"kubernetes.io/projected/64212cd6-8eb3-451b-9a4a-d89c567772bc-kube-api-access-sdb9t\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:17.728846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.728845    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775c5258-7f5d-4f58-ae04-f4449721275a-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:17.730775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.730753    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775c5258-7f5d-4f58-ae04-f4449721275a-kube-api-access-hmggw" pod "775c5258-7f5d-4f58-ae04-f4449721275a" (UID: "775c5258-7f5d-4f58-ae04-f4449721275a"). InnerVolumeSpecName "kube-api-access-hmggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:17.730968 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.730947    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75395ea0-7424-4cc5-a6a6-46150fc94050-kube-api-access-rfbtw" pod "75395ea0-7424-4cc5-a6a6-46150fc94050" (UID: "75395ea0-7424-4cc5-a6a6-46150fc94050"). InnerVolumeSpecName "kube-api-access-rfbtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:17.829718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.829685    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmggw\" (UniqueName: \"kubernetes.io/projected/775c5258-7f5d-4f58-ae04-f4449721275a-kube-api-access-hmggw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:17.829718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.829712    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfbtw\" (UniqueName: \"kubernetes.io/projected/75395ea0-7424-4cc5-a6a6-46150fc94050-kube-api-access-rfbtw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:17.894340 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.894270    5817 scope.go:122] "RemoveContainer" containerID="3a9a17c823bb6ca3899c297cee6111c3159174b28030ec5acb0e734fd02ee1fc"
Nov 03 11:30:17.960026 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:17.959986    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:18.271377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.271349    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"
Nov 03 11:30:18.271835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.271380    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw" event={"ID":"64212cd6-8eb3-451b-9a4a-d89c567772bc","Type":"ContainerDied","Data":"1949cbf9092628ab1d6aa2046bedde3428b64d7ab8fad5b18a2726326cef88f3"}
Nov 03 11:30:18.271835 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.271445    5817 scope.go:122] "RemoveContainer" containerID="3470510a5a41a87dc50206e912eb28489523f9fd2570aaabdc7f138de5e7a9ce"
Nov 03 11:30:18.273354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.273336    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"
Nov 03 11:30:18.273684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.273381    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm" event={"ID":"75395ea0-7424-4cc5-a6a6-46150fc94050","Type":"ContainerDied","Data":"fc6528803130fceb3357e65797b33dc9b12b6690e6c51e12c7cac2cf23fd33d0"}
Nov 03 11:30:18.279729 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.279712    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"
Nov 03 11:30:18.279843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.279758    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms" event={"ID":"775c5258-7f5d-4f58-ae04-f4449721275a","Type":"ContainerDied","Data":"ba54c2de33c2d464055fec574829ce0d448abc8e89081427c3f142d6e8f97032"}
Nov 03 11:30:18.281355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.281334    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"]
Nov 03 11:30:18.284627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.284147    5817 scope.go:122] "RemoveContainer" containerID="1e92a4f6ade28b4199995f58b527594671d2594bea5d7e1d8a0f43d6f409496c"
Nov 03 11:30:18.300479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.300460    5817 scope.go:122] "RemoveContainer" containerID="600a7061c8e3af9bad731ec46b21e3263cfc69f78a0ffb0ddc0e0f08ec93abc4"
Nov 03 11:30:18.308770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.308744    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"]
Nov 03 11:30:18.315202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.315176    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"]
Nov 03 11:30:18.321838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.321809    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895090-vxhgw"]
Nov 03 11:30:18.327792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.327769    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"]
Nov 03 11:30:18.341628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.339687    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-35936b-29895090-ff6jm"]
Nov 03 11:30:18.342997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.342971    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"]
Nov 03 11:30:18.362079 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.362046    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"]
Nov 03 11:30:18.436339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.436218    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"]
Nov 03 11:30:18.456233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.456195    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-836589-29895090-z6qms"]
Nov 03 11:30:18.533640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.533614    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:18.631344 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.631315    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:18.654084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.654048    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6629ec48-f9c3-4fbe-9678-5725518af1f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6629ec48-f9c3-4fbe-9678-5725518af1f2-maas-api-service-ca\") pod \"6629ec48-f9c3-4fbe-9678-5725518af1f2\" (UID: \"6629ec48-f9c3-4fbe-9678-5725518af1f2\") "
Nov 03 11:30:18.658217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.654659    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6629ec48-f9c3-4fbe-9678-5725518af1f2-maas-api-service-ca" pod "6629ec48-f9c3-4fbe-9678-5725518af1f2" (UID: "6629ec48-f9c3-4fbe-9678-5725518af1f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:18.661748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.659555    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:18.666106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.665687    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6629ec48-f9c3-4fbe-9678-5725518af1f2-kube-api-access-5mgp7\" (UniqueName: \"kubernetes.io/projected/6629ec48-f9c3-4fbe-9678-5725518af1f2-kube-api-access-5mgp7\") pod \"6629ec48-f9c3-4fbe-9678-5725518af1f2\" (UID: \"6629ec48-f9c3-4fbe-9678-5725518af1f2\") "
Nov 03 11:30:18.666708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.666541    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6629ec48-f9c3-4fbe-9678-5725518af1f2-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.704073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.675076    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6629ec48-f9c3-4fbe-9678-5725518af1f2-kube-api-access-5mgp7" pod "6629ec48-f9c3-4fbe-9678-5725518af1f2" (UID: "6629ec48-f9c3-4fbe-9678-5725518af1f2"). InnerVolumeSpecName "kube-api-access-5mgp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:18.717529 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.717501    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:18.736285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.736214    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:18.805037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.805001    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-maas-api-service-ca\") pod \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\" (UID: \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\") "
Nov 03 11:30:18.805228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.805078    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-kube-api-access-r9gc9\" (UniqueName: \"kubernetes.io/projected/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-kube-api-access-r9gc9\") pod \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\" (UID: \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\") "
Nov 03 11:30:18.805228 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.805131    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6876d8-0a8e-4a9f-a750-6d617e44f545-kube-api-access-9rnmm\" (UniqueName: \"kubernetes.io/projected/8d6876d8-0a8e-4a9f-a750-6d617e44f545-kube-api-access-9rnmm\") pod \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\" (UID: \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\") "
Nov 03 11:30:18.805395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.805225    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-kube-api-access-zjrpx\" (UniqueName: \"kubernetes.io/projected/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-kube-api-access-zjrpx\") pod \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\" (UID: \"b6b0fd1d-92bb-4031-846c-4e10c24c2b22\") "
Nov 03 11:30:18.805395 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.805341    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d6876d8-0a8e-4a9f-a750-6d617e44f545-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6876d8-0a8e-4a9f-a750-6d617e44f545-maas-api-service-ca\") pod \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\" (UID: \"8d6876d8-0a8e-4a9f-a750-6d617e44f545\") "
Nov 03 11:30:18.805516 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.805418    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-maas-api-service-ca\") pod \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\" (UID: \"5d50f1d2-4dca-4f89-beef-be956cbfdb2f\") "
Nov 03 11:30:18.806049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.806021    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mgp7\" (UniqueName: \"kubernetes.io/projected/6629ec48-f9c3-4fbe-9678-5725518af1f2-kube-api-access-5mgp7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.806331 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.806304    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-maas-api-service-ca" pod "b6b0fd1d-92bb-4031-846c-4e10c24c2b22" (UID: "b6b0fd1d-92bb-4031-846c-4e10c24c2b22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:18.806421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.806387    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-maas-api-service-ca" pod "5d50f1d2-4dca-4f89-beef-be956cbfdb2f" (UID: "5d50f1d2-4dca-4f89-beef-be956cbfdb2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:18.806775 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.806752    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6876d8-0a8e-4a9f-a750-6d617e44f545-maas-api-service-ca" pod "8d6876d8-0a8e-4a9f-a750-6d617e44f545" (UID: "8d6876d8-0a8e-4a9f-a750-6d617e44f545"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:18.809204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.809169    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-kube-api-access-zjrpx" pod "b6b0fd1d-92bb-4031-846c-4e10c24c2b22" (UID: "b6b0fd1d-92bb-4031-846c-4e10c24c2b22"). InnerVolumeSpecName "kube-api-access-zjrpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:18.809386 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.809341    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-kube-api-access-r9gc9" pod "5d50f1d2-4dca-4f89-beef-be956cbfdb2f" (UID: "5d50f1d2-4dca-4f89-beef-be956cbfdb2f"). InnerVolumeSpecName "kube-api-access-r9gc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:18.810027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.809990    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6876d8-0a8e-4a9f-a750-6d617e44f545-kube-api-access-9rnmm" pod "8d6876d8-0a8e-4a9f-a750-6d617e44f545" (UID: "8d6876d8-0a8e-4a9f-a750-6d617e44f545"). InnerVolumeSpecName "kube-api-access-9rnmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:18.895203 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.895177    5817 scope.go:122] "RemoveContainer" containerID="06aa5ac658e78b73f7ed1366b3893967e8b8ba83647a6d46e78adf5479dcd33d"
Nov 03 11:30:18.907389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.907365    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjrpx\" (UniqueName: \"kubernetes.io/projected/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-kube-api-access-zjrpx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.907389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.907391    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6876d8-0a8e-4a9f-a750-6d617e44f545-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.907527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.907401    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.907527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.907410    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0fd1d-92bb-4031-846c-4e10c24c2b22-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.907527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.907418    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9gc9\" (UniqueName: \"kubernetes.io/projected/5d50f1d2-4dca-4f89-beef-be956cbfdb2f-kube-api-access-r9gc9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.907527 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:18.907428    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rnmm\" (UniqueName: \"kubernetes.io/projected/8d6876d8-0a8e-4a9f-a750-6d617e44f545-kube-api-access-9rnmm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:19.293644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.293534    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"
Nov 03 11:30:19.293644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.293554    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt" event={"ID":"5d50f1d2-4dca-4f89-beef-be956cbfdb2f","Type":"ContainerDied","Data":"9451fbce726b488bd862a4cb0634d3fb5989b03a7855c3ccaf8ecf6992281c28"}
Nov 03 11:30:19.294202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.293656    5817 scope.go:122] "RemoveContainer" containerID="c77843cd5a1492db61626f01bc3873d162a47b4f9165b451a8af0a61531968ff"
Nov 03 11:30:19.296366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.296342    5817 generic.go:356] "Generic (PLEG): container finished" podID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerID="ca6ef66c268b254af31a8f63afea22d71902bada52c225d42790cf7bbf2f5f0d" exitCode=60
Nov 03 11:30:19.296479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.296451    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" event={"ID":"71b3de98-5049-4d08-a909-3d8311ec07ad","Type":"ContainerDied","Data":"ca6ef66c268b254af31a8f63afea22d71902bada52c225d42790cf7bbf2f5f0d"}
Nov 03 11:30:19.296926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.296905    5817 scope.go:122] "RemoveContainer" containerID="ca6ef66c268b254af31a8f63afea22d71902bada52c225d42790cf7bbf2f5f0d"
Nov 03 11:30:19.297256 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:19.297231    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7_odh-ai-gateway-infra(71b3de98-5049-4d08-a909-3d8311ec07ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad"
Nov 03 11:30:19.298385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.298355    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2" event={"ID":"b6b0fd1d-92bb-4031-846c-4e10c24c2b22","Type":"ContainerDied","Data":"87dae77454706e9b8e7d3a6e1d86714d1fbc041d3f207d3b98a823b8b21e64a3"}
Nov 03 11:30:19.298953 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.298363    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"
Nov 03 11:30:19.302089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.302067    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" event={"ID":"e44d4c32-eab6-45d8-879d-134c5ac585c2","Type":"ContainerStarted","Data":"4662d30f7e0a65c6cb1e873d679163f8c3d2f2cf071b12431b02ba00443e3292"}
Nov 03 11:30:19.307145 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.307125    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"
Nov 03 11:30:19.307303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.307148    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh" event={"ID":"6629ec48-f9c3-4fbe-9678-5725518af1f2","Type":"ContainerDied","Data":"db969074c7461923b685c5c3e6d1818ad3681aac2e4381929e410d6555d507ad"}
Nov 03 11:30:19.308720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.308702    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm" event={"ID":"8d6876d8-0a8e-4a9f-a750-6d617e44f545","Type":"ContainerDied","Data":"2d672f19b05c1a7dce5537621eefbbdf9657a6a171af52163f6f6b429f174b08"}
Nov 03 11:30:19.308910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.308790    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"
Nov 03 11:30:19.311488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.310688    5817 scope.go:122] "RemoveContainer" containerID="3a9a17c823bb6ca3899c297cee6111c3159174b28030ec5acb0e734fd02ee1fc"
Nov 03 11:30:19.328535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.328504    5817 scope.go:122] "RemoveContainer" containerID="04f4f708fbe82f6857a9e2515948aa4ce35194be815ae3ed30068dd228ef8440"
Nov 03 11:30:19.341102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.341046    5817 scope.go:122] "RemoveContainer" containerID="3d099895bf0dc94677baf1e926e6225161d0d02424c3f7f9436ca18070859efc"
Nov 03 11:30:19.351932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.351917    5817 scope.go:122] "RemoveContainer" containerID="807395b582e452af4829bf9e30813e39729c7c468b44bb49f0b4d5be58a9b320"
Nov 03 11:30:19.353677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.353655    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"]
Nov 03 11:30:19.364149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.364128    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895090-j44mm"]
Nov 03 11:30:19.376870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.376851    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"]
Nov 03 11:30:19.386517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.386498    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895090-4ptjh"]
Nov 03 11:30:19.400519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.400500    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"]
Nov 03 11:30:19.404343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.404323    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb5766-29895090-qx4t2"]
Nov 03 11:30:19.413294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.413273    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"]
Nov 03 11:30:19.418417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.418396    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895090-568qt"]
Nov 03 11:30:19.566653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.560423    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:19.846033 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:19.845953    5817 logging.go:55] [core] [Channel #18693 SubChannel #18694]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.64:50052", ServerName: "10.133.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.64:50052: connect: connection refused"
Nov 03 11:30:19.930712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.930672    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" path="/var/lib/kubelet/pods/5d50f1d2-4dca-4f89-beef-be956cbfdb2f/volumes"
Nov 03 11:30:19.931317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.931294    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" path="/var/lib/kubelet/pods/64212cd6-8eb3-451b-9a4a-d89c567772bc/volumes"
Nov 03 11:30:19.932184 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.932164    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" path="/var/lib/kubelet/pods/6629ec48-f9c3-4fbe-9678-5725518af1f2/volumes"
Nov 03 11:30:19.933300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.933280    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" path="/var/lib/kubelet/pods/75395ea0-7424-4cc5-a6a6-46150fc94050/volumes"
Nov 03 11:30:19.934160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.933977    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" path="/var/lib/kubelet/pods/775c5258-7f5d-4f58-ae04-f4449721275a/volumes"
Nov 03 11:30:19.934840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.934820    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" path="/var/lib/kubelet/pods/8d6876d8-0a8e-4a9f-a750-6d617e44f545/volumes"
Nov 03 11:30:19.936549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.935972    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" path="/var/lib/kubelet/pods/b6b0fd1d-92bb-4031-846c-4e10c24c2b22/volumes"
Nov 03 11:30:19.979191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:19.979167    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 11:30:20.022332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.022301    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-plugins-config-volume\") pod \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\" (UID: \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\") "
Nov 03 11:30:20.022753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.022729    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-plugins-config-volume" pod "1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" (UID: "1f128c57-8e1f-4ee4-8ab3-2b031ba7d388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:20.022854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.022837    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-kube-api-access-x9m79\" (UniqueName: \"kubernetes.io/projected/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-kube-api-access-x9m79\") pod \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\" (UID: \"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388\") "
Nov 03 11:30:20.023565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.023520    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.035668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.032000    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-kube-api-access-x9m79" pod "1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" (UID: "1f128c57-8e1f-4ee4-8ab3-2b031ba7d388"). InnerVolumeSpecName "kube-api-access-x9m79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:20.126230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.126147    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9m79\" (UniqueName: \"kubernetes.io/projected/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388-kube-api-access-x9m79\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.323895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.323862    5817 generic.go:356] "Generic (PLEG): container finished" podID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" containerID="135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32" exitCode=0
Nov 03 11:30:20.324502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.323904    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"
Nov 03 11:30:20.324502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.323951    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" event={"ID":"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388","Type":"ContainerDied","Data":"135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32"}
Nov 03 11:30:20.324502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.324009    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" event={"ID":"1f128c57-8e1f-4ee4-8ab3-2b031ba7d388","Type":"ContainerDied","Data":"3d87e2610dcf2086d355a1d6996f89fe95f66180af31290c33fd2b714db7e7c2"}
Nov 03 11:30:20.324502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.324036    5817 scope.go:122] "RemoveContainer" containerID="135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32"
Nov 03 11:30:20.332621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.332551    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"]
Nov 03 11:30:20.336144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.336115    5817 generic.go:356] "Generic (PLEG): container finished" podID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerID="4662d30f7e0a65c6cb1e873d679163f8c3d2f2cf071b12431b02ba00443e3292" exitCode=60
Nov 03 11:30:20.336261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.336186    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" event={"ID":"e44d4c32-eab6-45d8-879d-134c5ac585c2","Type":"ContainerDied","Data":"4662d30f7e0a65c6cb1e873d679163f8c3d2f2cf071b12431b02ba00443e3292"}
Nov 03 11:30:20.336618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.336561    5817 scope.go:122] "RemoveContainer" containerID="4662d30f7e0a65c6cb1e873d679163f8c3d2f2cf071b12431b02ba00443e3292"
Nov 03 11:30:20.336932 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:20.336889    5817 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5_odh-ai-gateway-infra(e44d4c32-eab6-45d8-879d-134c5ac585c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2"
Nov 03 11:30:20.339188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.339170    5817 scope.go:122] "RemoveContainer" containerID="135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32"
Nov 03 11:30:20.339436 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:20.339413    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32\": container with ID starting with 135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32 not found: ID does not exist" containerID="135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32"
Nov 03 11:30:20.339511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.339442    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32"} err="failed to get container status \"135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32\": rpc error: code = NotFound desc = could not find container \"135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32\": container with ID starting with 135f57bf45c21775ef0bf532dd5a7ecb21fe98df62a0d371de0099df9c673d32 not found: ID does not exist"
Nov 03 11:30:20.339511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.339457    5817 scope.go:122] "RemoveContainer" containerID="06aa5ac658e78b73f7ed1366b3893967e8b8ba83647a6d46e78adf5479dcd33d"
Nov 03 11:30:20.369609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.362250    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"]
Nov 03 11:30:20.388959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.388873    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:20.423002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.417463    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"]
Nov 03 11:30:20.447995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.447956    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g"]
Nov 03 11:30:20.574870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.574847    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:20.610474 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.610434    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp" podUID="0b55d88b-58c7-48ab-b1ee-5016cb9dc329" containerName="payload-processing" containerID="cri-o://9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa" gracePeriod=25
Nov 03 11:30:20.644372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.644304    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b3de98-5049-4d08-a909-3d8311ec07ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b3de98-5049-4d08-a909-3d8311ec07ad-maas-api-service-ca\") pod \"71b3de98-5049-4d08-a909-3d8311ec07ad\" (UID: \"71b3de98-5049-4d08-a909-3d8311ec07ad\") "
Nov 03 11:30:20.644510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.644459    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b3de98-5049-4d08-a909-3d8311ec07ad-kube-api-access-gkxrc\" (UniqueName: \"kubernetes.io/projected/71b3de98-5049-4d08-a909-3d8311ec07ad-kube-api-access-gkxrc\") pod \"71b3de98-5049-4d08-a909-3d8311ec07ad\" (UID: \"71b3de98-5049-4d08-a909-3d8311ec07ad\") "
Nov 03 11:30:20.644734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.644711    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b3de98-5049-4d08-a909-3d8311ec07ad-maas-api-service-ca" pod "71b3de98-5049-4d08-a909-3d8311ec07ad" (UID: "71b3de98-5049-4d08-a909-3d8311ec07ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:20.645021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.644998    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b3de98-5049-4d08-a909-3d8311ec07ad-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.646789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.646764    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b3de98-5049-4d08-a909-3d8311ec07ad-kube-api-access-gkxrc" pod "71b3de98-5049-4d08-a909-3d8311ec07ad" (UID: "71b3de98-5049-4d08-a909-3d8311ec07ad"). InnerVolumeSpecName "kube-api-access-gkxrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:20.745848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.745824    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkxrc\" (UniqueName: \"kubernetes.io/projected/71b3de98-5049-4d08-a909-3d8311ec07ad-kube-api-access-gkxrc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.845979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:20.845941    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d5d3c3-748448dc75-dv65g" podUID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.64:50052\" within 1s: context deadline exceeded"
Nov 03 11:30:21.101711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.101669    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:21.346920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.346895    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"
Nov 03 11:30:21.347378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.346922    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7" event={"ID":"71b3de98-5049-4d08-a909-3d8311ec07ad","Type":"ContainerDied","Data":"39150d55df765962890ceb58c45375abff1893caefd6e337f8658406e2bff8a4"}
Nov 03 11:30:21.347378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.346991    5817 scope.go:122] "RemoveContainer" containerID="ca6ef66c268b254af31a8f63afea22d71902bada52c225d42790cf7bbf2f5f0d"
Nov 03 11:30:21.384820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.384794    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"]
Nov 03 11:30:21.391515 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.391484    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895090-kl7q7"]
Nov 03 11:30:21.533852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.533826    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:21.694611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.687086    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44d4c32-eab6-45d8-879d-134c5ac585c2-kube-api-access-j2v2c\" (UniqueName: \"kubernetes.io/projected/e44d4c32-eab6-45d8-879d-134c5ac585c2-kube-api-access-j2v2c\") pod \"e44d4c32-eab6-45d8-879d-134c5ac585c2\" (UID: \"e44d4c32-eab6-45d8-879d-134c5ac585c2\") "
Nov 03 11:30:21.694611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.694041    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:21.701125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.695116    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44d4c32-eab6-45d8-879d-134c5ac585c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d4c32-eab6-45d8-879d-134c5ac585c2-maas-api-service-ca\") pod \"e44d4c32-eab6-45d8-879d-134c5ac585c2\" (UID: \"e44d4c32-eab6-45d8-879d-134c5ac585c2\") "
Nov 03 11:30:21.707609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.701803    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44d4c32-eab6-45d8-879d-134c5ac585c2-maas-api-service-ca" pod "e44d4c32-eab6-45d8-879d-134c5ac585c2" (UID: "e44d4c32-eab6-45d8-879d-134c5ac585c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:21.720671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.720633    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44d4c32-eab6-45d8-879d-134c5ac585c2-kube-api-access-j2v2c" pod "e44d4c32-eab6-45d8-879d-134c5ac585c2" (UID: "e44d4c32-eab6-45d8-879d-134c5ac585c2"). InnerVolumeSpecName "kube-api-access-j2v2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:21.797896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.797859    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44d4c32-eab6-45d8-879d-134c5ac585c2-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:21.797896 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.797892    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2v2c\" (UniqueName: \"kubernetes.io/projected/e44d4c32-eab6-45d8-879d-134c5ac585c2-kube-api-access-j2v2c\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:21.905747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.905713    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" path="/var/lib/kubelet/pods/1f128c57-8e1f-4ee4-8ab3-2b031ba7d388/volumes"
Nov 03 11:30:21.906325 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:21.906305    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" path="/var/lib/kubelet/pods/71b3de98-5049-4d08-a909-3d8311ec07ad/volumes"
Nov 03 11:30:22.333640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.333601    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:22.388003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.387815    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5" event={"ID":"e44d4c32-eab6-45d8-879d-134c5ac585c2","Type":"ContainerDied","Data":"f1026007ad311b8d58da06a84c3793bc6ff3c376c42edc951cb13fcbbeb76a82"}
Nov 03 11:30:22.388003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.387921    5817 scope.go:122] "RemoveContainer" containerID="4662d30f7e0a65c6cb1e873d679163f8c3d2f2cf071b12431b02ba00443e3292"
Nov 03 11:30:22.388688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.388661    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"
Nov 03 11:30:22.410955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.410903    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"]
Nov 03 11:30:22.417067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.417038    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895090-rjkb5"]
Nov 03 11:30:22.428614 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:22.428554    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:30:22.482964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.482938    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:22.593533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.593307    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"]
Nov 03 11:30:22.593753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.593604    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" containerID="cri-o://07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4" gracePeriod=30
Nov 03 11:30:22.660637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:22.660598    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:23.233618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:23.228206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:23.902831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:23.902797    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" path="/var/lib/kubelet/pods/e44d4c32-eab6-45d8-879d-134c5ac585c2/volumes"
Nov 03 11:30:23.950657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:23.948634    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:24.083876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:24.083844    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:24.577045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:24.572180    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:25.128547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.128511    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:25.286599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.271487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:25.458056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.458027    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"]
Nov 03 11:30:25.461565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.461542    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-185908-5q9s7"]
Nov 03 11:30:25.505454 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.505389    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:25.707340 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.706444    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:25.852384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.852309    5817 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:30:25.852384 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.852364    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:30:25.902702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:25.902674    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19875dc4-8c13-4271-95c6-c03df5861569" path="/var/lib/kubelet/pods/19875dc4-8c13-4271-95c6-c03df5861569/volumes"
Nov 03 11:30:26.125462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:26.125366    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:26.330852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:26.330819    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:26.481266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:26.481233    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:26.681657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:26.681616    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:27.273599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:27.272797    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:27.381583 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:27.381547    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp" podUID="0b55d88b-58c7-48ab-b1ee-5016cb9dc329" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:30:27.882278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:27.882217    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 11:30:27.953260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:27.953221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:28.033075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.033035    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"]
Nov 03 11:30:28.037118 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.037092    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2779423d-d8p8l"]
Nov 03 11:30:28.041791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.041764    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-credential-socket\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.041956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.041889    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istiod-ca-cert\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.042037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.041982    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-envoy\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.042037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042019    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-certs\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.042171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042041    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-credential-socket" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:28.042171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042059    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-data\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.042284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042186    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-podinfo\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.042284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042226    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-kube-api-access-mgdt7\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-kube-api-access-mgdt7\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.042284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042268    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-socket\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.042467 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042304    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-token\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-token\") pod \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\" (UID: \"e06ebb7c-9841-4fb0-a7d5-8f945814e368\") "
Nov 03 11:30:28.042607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042543    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-certs" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:28.042935 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.042914    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-data" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:28.043424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.043402    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.043823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.043796    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.044974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.044952    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-socket" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:28.045075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.045005    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-podinfo" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:30:28.045226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.045204    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-envoy" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:28.045504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.045487    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istiod-ca-cert" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:28.047109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.047086    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-kube-api-access-mgdt7" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "kube-api-access-mgdt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:28.047338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.047318    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-token" pod "e06ebb7c-9841-4fb0-a7d5-8f945814e368" (UID: "e06ebb7c-9841-4fb0-a7d5-8f945814e368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:28.145201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.145121    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.145201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.145149    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.145201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.145158    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.145201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.145167    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgdt7\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-kube-api-access-mgdt7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.145201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.145178    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06ebb7c-9841-4fb0-a7d5-8f945814e368-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.145201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.145186    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.145201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.145195    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06ebb7c-9841-4fb0-a7d5-8f945814e368-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:28.383286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.383246    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:28.446017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.445989    5817 generic.go:356] "Generic (PLEG): container finished" podID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerID="07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4" exitCode=0
Nov 03 11:30:28.446188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.446025    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" event={"ID":"e06ebb7c-9841-4fb0-a7d5-8f945814e368","Type":"ContainerDied","Data":"07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4"}
Nov 03 11:30:28.446188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.446042    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"
Nov 03 11:30:28.446188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.446073    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5" event={"ID":"e06ebb7c-9841-4fb0-a7d5-8f945814e368","Type":"ContainerDied","Data":"d3bc2300f5e6101c75a8a41d63cd60d3e26b72bcec49d87aee6d22ba19bd79b1"}
Nov 03 11:30:28.446188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.446099    5817 scope.go:122] "RemoveContainer" containerID="07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4"
Nov 03 11:30:28.458337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.458317    5817 scope.go:122] "RemoveContainer" containerID="07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4"
Nov 03 11:30:28.458645 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:28.458622    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4\": container with ID starting with 07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4 not found: ID does not exist" containerID="07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4"
Nov 03 11:30:28.458708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.458657    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4"} err="failed to get container status \"07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4\": rpc error: code = NotFound desc = could not find container \"07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4\": container with ID starting with 07534a2923f9627131325d65aec86fe1ec784e9f6958a0543ebe75b8c374e3d4 not found: ID does not exist"
Nov 03 11:30:28.480914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.480820    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"]
Nov 03 11:30:28.491258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.491232    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-5kxd5"]
Nov 03 11:30:28.707385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:28.707298    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:29.394645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:29.394546    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:29.901951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:29.901919    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" path="/var/lib/kubelet/pods/e06ebb7c-9841-4fb0-a7d5-8f945814e368/volumes"
Nov 03 11:30:29.902360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:29.902346    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5ba865-e06b-42d3-ba05-fc5093f54f32" path="/var/lib/kubelet/pods/ee5ba865-e06b-42d3-ba05-fc5093f54f32/volumes"
Nov 03 11:30:30.054030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:30.053996    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:30.851219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:30.851188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:31.563476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:31.563406    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:32.187261 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:32.187229    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:32.336270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:32.336210    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:32.450566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:32.450484    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"]
Nov 03 11:30:32.450772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:32.450738    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy" containerID="cri-o://3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b" gracePeriod=30
Nov 03 11:30:32.536558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:32.536517    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:33.118093 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.118058    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"]
Nov 03 11:30:33.118984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.118964    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:33.119067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.118986    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:33.119067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119003    5817 cpu_manager.go:414] "containerMap: removing container" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.119067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119012    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.119067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119026    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:33.119067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119033    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:33.119067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119047    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:33.119067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119054    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:33.119067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119066    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119073    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119082    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119090    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119099    5817 cpu_manager.go:414] "containerMap: removing container" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119106    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119119    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119125    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119145    5817 cpu_manager.go:414] "containerMap: removing container" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119151    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119168    5817 cpu_manager.go:414] "containerMap: removing container" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119175    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119187    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119194    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119207    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119214    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119227    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119234    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119247    5817 cpu_manager.go:414] "containerMap: removing container" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119253    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119270    5817 cpu_manager.go:414] "containerMap: removing container" podUID="19875dc4-8c13-4271-95c6-c03df5861569" containerName="revoke-keys"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119276    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19875dc4-8c13-4271-95c6-c03df5861569" containerName="revoke-keys"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119286    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119291    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119306    5817 cpu_manager.go:414] "containerMap: removing container" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119313    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119324    5817 cpu_manager.go:414] "containerMap: removing container" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119330    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119344    5817 cpu_manager.go:414] "containerMap: removing container" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119348    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119355    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:33.119346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119359    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119372    5817 cpu_manager.go:414] "containerMap: removing container" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119379    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119395    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119401    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119413    5817 cpu_manager.go:414] "containerMap: removing container" podUID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" containerName="payload-processing"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119420    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" containerName="payload-processing"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119431    5817 cpu_manager.go:414] "containerMap: removing container" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119437    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119448    5817 cpu_manager.go:414] "containerMap: removing container" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119457    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119620    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119636    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119645    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119656    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119666    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119676    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119687    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119695    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119705    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f128c57-8e1f-4ee4-8ab3-2b031ba7d388" containerName="payload-processing"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119714    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119721    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b3de98-5049-4d08-a909-3d8311ec07ad" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119731    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119739    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119746    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119754    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119764    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119773    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119782    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06ebb7c-9841-4fb0-a7d5-8f945814e368" containerName="istio-proxy"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119792    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119803    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119812    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="775c5258-7f5d-4f58-ae04-f4449721275a" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119821    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119828    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119837    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119846    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:33.120337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.119857    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="19875dc4-8c13-4271-95c6-c03df5861569" containerName="revoke-keys"
Nov 03 11:30:33.123836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.123814    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.130149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.130095    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"]
Nov 03 11:30:33.152594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.131715    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:30:33.152594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.131756    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:30:33.226720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.226685    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qf9w\" (UniqueName: \"kubernetes.io/projected/56ad61ab-2070-4932-923e-a167ed8c2ddf-kube-api-access-6qf9w\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.227095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.226736    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.227095 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.226867    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.259246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.259198    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:33.328312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.328279    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qf9w\" (UniqueName: \"kubernetes.io/projected/56ad61ab-2070-4932-923e-a167ed8c2ddf-kube-api-access-6qf9w\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.328510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.328322    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.328510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.328403    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.329169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.329144    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.330975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.330947    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.336445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.336418    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qf9w\" (UniqueName: \"kubernetes.io/projected/56ad61ab-2070-4932-923e-a167ed8c2ddf-kube-api-access-6qf9w\") pod \"maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.491618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.491592    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:33.780364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:33.780335    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"]
Nov 03 11:30:33.784892 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:33.784867    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ad61ab_2070_4932_923e_a167ed8c2ddf.slice/crio-b1889719e8100c242dd3cff8200bba4c1d601af8657871210766189068301bc8 WatchSource:0}: Error finding container b1889719e8100c242dd3cff8200bba4c1d601af8657871210766189068301bc8: Status 404 returned error can't find the container with id b1889719e8100c242dd3cff8200bba4c1d601af8657871210766189068301bc8
Nov 03 11:30:34.000546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:34.000468    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:34.353523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:34.353438    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:34.509586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:34.509529    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn" event={"ID":"56ad61ab-2070-4932-923e-a167ed8c2ddf","Type":"ContainerStarted","Data":"e3061e3c41fcf11c734f7d210790801688594f6105b4d7856d71c6f4e06afc60"}
Nov 03 11:30:34.509752 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:34.509611    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn" event={"ID":"56ad61ab-2070-4932-923e-a167ed8c2ddf","Type":"ContainerStarted","Data":"b1889719e8100c242dd3cff8200bba4c1d601af8657871210766189068301bc8"}
Nov 03 11:30:34.530542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:34.530482    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn" podStartSLOduration=1.530465232 podStartE2EDuration="1.530465232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:34.524442796 +0000 UTC m=+3664.730024846" watchObservedRunningTime="2026-11-03 11:30:34.530465232 +0000 UTC m=+3664.736047290"
Nov 03 11:30:34.660649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:34.649062    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:35.427262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:35.425698    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:35.522005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:35.521963    5817 generic.go:356] "Generic (PLEG): container finished" podID="56ad61ab-2070-4932-923e-a167ed8c2ddf" containerID="e3061e3c41fcf11c734f7d210790801688594f6105b4d7856d71c6f4e06afc60" exitCode=0
Nov 03 11:30:35.522176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:35.522037    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn" event={"ID":"56ad61ab-2070-4932-923e-a167ed8c2ddf","Type":"ContainerDied","Data":"e3061e3c41fcf11c734f7d210790801688594f6105b4d7856d71c6f4e06afc60"}
Nov 03 11:30:35.914109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:35.914080    5817 scope.go:122] "RemoveContainer" containerID="3d3f0eca28260aebd96b6b66e96442b98c69e6a5ca463fd84527d401cf3967a9"
Nov 03 11:30:36.053186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.053158    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:36.699533 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.699498    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:36.773615 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.772236    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:36.781313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.781269    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ad61ab-2070-4932-923e-a167ed8c2ddf-kube-api-access-6qf9w\" (UniqueName: \"kubernetes.io/projected/56ad61ab-2070-4932-923e-a167ed8c2ddf-kube-api-access-6qf9w\") pod \"56ad61ab-2070-4932-923e-a167ed8c2ddf\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") "
Nov 03 11:30:36.781528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.781475    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-xks-ca\") pod \"56ad61ab-2070-4932-923e-a167ed8c2ddf\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") "
Nov 03 11:30:36.781528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.781501    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-openshift-ca\") pod \"56ad61ab-2070-4932-923e-a167ed8c2ddf\" (UID: \"56ad61ab-2070-4932-923e-a167ed8c2ddf\") "
Nov 03 11:30:36.783337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.782871    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-openshift-ca" pod "56ad61ab-2070-4932-923e-a167ed8c2ddf" (UID: "56ad61ab-2070-4932-923e-a167ed8c2ddf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:36.785270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.785205    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-xks-ca" pod "56ad61ab-2070-4932-923e-a167ed8c2ddf" (UID: "56ad61ab-2070-4932-923e-a167ed8c2ddf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:36.804458 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.798100    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ad61ab-2070-4932-923e-a167ed8c2ddf-kube-api-access-6qf9w" pod "56ad61ab-2070-4932-923e-a167ed8c2ddf" (UID: "56ad61ab-2070-4932-923e-a167ed8c2ddf"). InnerVolumeSpecName "kube-api-access-6qf9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:36.883040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.883010    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:36.883040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.883040    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ad61ab-2070-4932-923e-a167ed8c2ddf-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:36.883224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:36.883051    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qf9w\" (UniqueName: \"kubernetes.io/projected/56ad61ab-2070-4932-923e-a167ed8c2ddf-kube-api-access-6qf9w\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:37.290648 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.290615    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:30:37.381622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.381588    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp" podUID="0b55d88b-58c7-48ab-b1ee-5016cb9dc329" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:30:37.476742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.476705    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:37.545489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.545414    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn" event={"ID":"56ad61ab-2070-4932-923e-a167ed8c2ddf","Type":"ContainerDied","Data":"b1889719e8100c242dd3cff8200bba4c1d601af8657871210766189068301bc8"}
Nov 03 11:30:37.545489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.545442    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"
Nov 03 11:30:37.545489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.545463    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1889719e8100c242dd3cff8200bba4c1d601af8657871210766189068301bc8"
Nov 03 11:30:37.766297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.766275    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.902799    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-token\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-token\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.902859    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-certs\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.902891    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-podinfo\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.902986    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-socket\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.903061    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-data\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.903158    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-kube-api-access-q4nvf\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-kube-api-access-q4nvf\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.903200    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033ade0f-f138-4a4b-b2b8-41d02673adab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033ade0f-f138-4a4b-b2b8-41d02673adab-istiod-ca-cert\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.903325    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-credential-socket\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.913649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.903372    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-envoy\") pod \"033ade0f-f138-4a4b-b2b8-41d02673adab\" (UID: \"033ade0f-f138-4a4b-b2b8-41d02673adab\") "
Nov 03 11:30:37.914296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.913835    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-socket" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:37.922596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.914600    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-certs" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:37.922596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.914895    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033ade0f-f138-4a4b-b2b8-41d02673adab-istiod-ca-cert" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:37.922596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.915126    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-credential-socket" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:37.922596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.921687    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-data" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:37.964691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.963004    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-kube-api-access-q4nvf" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "kube-api-access-q4nvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:37.968592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.965236    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-podinfo" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:30:37.969600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.968774    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-token" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:37.969600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:37.968897    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-envoy" pod "033ade0f-f138-4a4b-b2b8-41d02673adab" (UID: "033ade0f-f138-4a4b-b2b8-41d02673adab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005643    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005679    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005714    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4nvf\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-kube-api-access-q4nvf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005728    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033ade0f-f138-4a4b-b2b8-41d02673adab-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005743    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005757    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005770    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005784    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033ade0f-f138-4a4b-b2b8-41d02673adab-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.006409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.005797    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033ade0f-f138-4a4b-b2b8-41d02673adab-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:38.376599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.374444    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:38.555799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.555758    5817 generic.go:356] "Generic (PLEG): container finished" podID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerID="3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b" exitCode=0
Nov 03 11:30:38.555993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.555814    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"
Nov 03 11:30:38.555993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.555882    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" event={"ID":"033ade0f-f138-4a4b-b2b8-41d02673adab","Type":"ContainerDied","Data":"3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b"}
Nov 03 11:30:38.555993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.555949    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h" event={"ID":"033ade0f-f138-4a4b-b2b8-41d02673adab","Type":"ContainerDied","Data":"fed679d667487be19e28dad7a8de439c1e61da3f7aad6553b2094f3587431587"}
Nov 03 11:30:38.555993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.555985    5817 scope.go:122] "RemoveContainer" containerID="3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b"
Nov 03 11:30:38.568547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.568528    5817 scope.go:122] "RemoveContainer" containerID="2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b"
Nov 03 11:30:38.578981 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.578750    5817 scope.go:122] "RemoveContainer" containerID="3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b"
Nov 03 11:30:38.579104 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:38.579045    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b\": container with ID starting with 3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b not found: ID does not exist" containerID="3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b"
Nov 03 11:30:38.579222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.579099    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b"} err="failed to get container status \"3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b\": rpc error: code = NotFound desc = could not find container \"3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b\": container with ID starting with 3f90b25d2a6570006ec8822832cdf84da7961f391354456c39ba36204adc297b not found: ID does not exist"
Nov 03 11:30:38.579222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.579122    5817 scope.go:122] "RemoveContainer" containerID="2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b"
Nov 03 11:30:38.579553 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:38.579513    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b\": container with ID starting with 2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b not found: ID does not exist" containerID="2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b"
Nov 03 11:30:38.580018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.579560    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b"} err="failed to get container status \"2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b\": rpc error: code = NotFound desc = could not find container \"2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b\": container with ID starting with 2e0c5af9f58e2f597929d556106577d803c2ca55f4724bce6cd8d43ecf83a72b not found: ID does not exist"
Nov 03 11:30:38.584169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.584150    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"]
Nov 03 11:30:38.589504 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:38.589483    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-185908-openshift-default-795bb9d77-cgh2h"]
Nov 03 11:30:39.197296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:39.193821    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:39.903726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:39.903695    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" path="/var/lib/kubelet/pods/033ade0f-f138-4a4b-b2b8-41d02673adab/volumes"
Nov 03 11:30:40.055443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:40.050445    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:40.672762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:40.672469    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:40.896088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:40.896065    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:30:41.046665 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.046635    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-plugins-config-volume\") pod \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\" (UID: \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\") "
Nov 03 11:30:41.046867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.046736    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-kube-api-access-4kpt2\" (UniqueName: \"kubernetes.io/projected/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-kube-api-access-4kpt2\") pod \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\" (UID: \"0b55d88b-58c7-48ab-b1ee-5016cb9dc329\") "
Nov 03 11:30:41.047406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.047104    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-plugins-config-volume" pod "0b55d88b-58c7-48ab-b1ee-5016cb9dc329" (UID: "0b55d88b-58c7-48ab-b1ee-5016cb9dc329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:41.047537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.047454    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:41.049309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.049286    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-kube-api-access-4kpt2" pod "0b55d88b-58c7-48ab-b1ee-5016cb9dc329" (UID: "0b55d88b-58c7-48ab-b1ee-5016cb9dc329"). InnerVolumeSpecName "kube-api-access-4kpt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:41.148641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.148530    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kpt2\" (UniqueName: \"kubernetes.io/projected/0b55d88b-58c7-48ab-b1ee-5016cb9dc329-kube-api-access-4kpt2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:41.380901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.375040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:41.586466 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.586429    5817 generic.go:356] "Generic (PLEG): container finished" podID="0b55d88b-58c7-48ab-b1ee-5016cb9dc329" containerID="9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa" exitCode=0
Nov 03 11:30:41.586748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.586504    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"
Nov 03 11:30:41.586748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.586519    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp" event={"ID":"0b55d88b-58c7-48ab-b1ee-5016cb9dc329","Type":"ContainerDied","Data":"9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa"}
Nov 03 11:30:41.586748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.586631    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp" event={"ID":"0b55d88b-58c7-48ab-b1ee-5016cb9dc329","Type":"ContainerDied","Data":"a96d433410b1060ef5209a719a51d6e4f8ae09d80ed114a9d069cc7aa877b2b3"}
Nov 03 11:30:41.586748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.586667    5817 scope.go:122] "RemoveContainer" containerID="9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa"
Nov 03 11:30:41.600323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.599981    5817 scope.go:122] "RemoveContainer" containerID="9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa"
Nov 03 11:30:41.600438 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:41.600323    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa\": container with ID starting with 9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa not found: ID does not exist" containerID="9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa"
Nov 03 11:30:41.600438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.600361    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa"} err="failed to get container status \"9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa\": rpc error: code = NotFound desc = could not find container \"9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa\": container with ID starting with 9aea830c1011d0db4f8828fa5eab2270ffd47785ed3903dcc7d93ea91529cfaa not found: ID does not exist"
Nov 03 11:30:41.612377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.612356    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"]
Nov 03 11:30:41.618445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.618424    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-185908-5f495fdb69-s9nbp"]
Nov 03 11:30:41.902010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:41.901938    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b55d88b-58c7-48ab-b1ee-5016cb9dc329" path="/var/lib/kubelet/pods/0b55d88b-58c7-48ab-b1ee-5016cb9dc329/volumes"
Nov 03 11:30:42.057041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:42.052652    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"]
Nov 03 11:30:42.057041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:42.053087    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:42.057041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:42.054245    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53303d76-q9jfs"]
Nov 03 11:30:42.647366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:42.647331    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:43.265612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:43.258261    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:43.397390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:43.397359    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:43.903077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:43.903041    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e7f6bd-590f-4fe8-9b0a-0125a282f6c0" path="/var/lib/kubelet/pods/01e7f6bd-590f-4fe8-9b0a-0125a282f6c0/volumes"
Nov 03 11:30:44.079809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.078563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:44.146085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146056    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"]
Nov 03 11:30:44.146654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146641    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146655    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146672    5817 cpu_manager.go:414] "containerMap: removing container" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146677    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75395ea0-7424-4cc5-a6a6-46150fc94050" containerName="cleanup"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146688    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0b55d88b-58c7-48ab-b1ee-5016cb9dc329" containerName="payload-processing"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146695    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b55d88b-58c7-48ab-b1ee-5016cb9dc329" containerName="payload-processing"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146703    5817 cpu_manager.go:414] "containerMap: removing container" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146709    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146715    5817 cpu_manager.go:414] "containerMap: removing container" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146720    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy"
Nov 03 11:30:44.146731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146732    5817 cpu_manager.go:414] "containerMap: removing container" podUID="56ad61ab-2070-4932-923e-a167ed8c2ddf" containerName="revoke-keys"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146736    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ad61ab-2070-4932-923e-a167ed8c2ddf" containerName="revoke-keys"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146751    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146756    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d50f1d2-4dca-4f89-beef-be956cbfdb2f" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146762    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146766    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b0fd1d-92bb-4031-846c-4e10c24c2b22" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146772    5817 cpu_manager.go:414] "containerMap: removing container" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146776    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146870    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b55d88b-58c7-48ab-b1ee-5016cb9dc329" containerName="payload-processing"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146876    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146882    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="033ade0f-f138-4a4b-b2b8-41d02673adab" containerName="istio-proxy"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146889    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ad61ab-2070-4932-923e-a167ed8c2ddf" containerName="revoke-keys"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146894    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44d4c32-eab6-45d8-879d-134c5ac585c2" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146904    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6876d8-0a8e-4a9f-a750-6d617e44f545" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146909    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6629ec48-f9c3-4fbe-9678-5725518af1f2" containerName="cleanup"
Nov 03 11:30:44.147051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.146915    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="64212cd6-8eb3-451b-9a4a-d89c567772bc" containerName="cleanup"
Nov 03 11:30:44.152513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.152442    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.156730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.156667    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"]
Nov 03 11:30:44.156874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.156857    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:30:44.156979 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.156966    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:30:44.316087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.316048    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j47f\" (UniqueName: \"kubernetes.io/projected/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-kube-api-access-6j47f\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.316605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.316102    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.316605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.316235    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.417479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.417349    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j47f\" (UniqueName: \"kubernetes.io/projected/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-kube-api-access-6j47f\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.417479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.417417    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.417810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.417508    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.418376 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.418345    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.420631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.420606    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.425688 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.425633    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j47f\" (UniqueName: \"kubernetes.io/projected/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-kube-api-access-6j47f\") pod \"maas-api-revoke-sub-00534afc-svm9x\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.468890 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.468861    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:44.769930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.769898    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"]
Nov 03 11:30:44.770344 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:44.770312    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b130fdf_59ac_4f66_ad7c_2a04636e1dd3.slice/crio-6110f4818b36c002b1bbbbb73998b30f0ae4e4464fb2763f137059d754da2569 WatchSource:0}: Error finding container 6110f4818b36c002b1bbbbb73998b30f0ae4e4464fb2763f137059d754da2569: Status 404 returned error can't find the container with id 6110f4818b36c002b1bbbbb73998b30f0ae4e4464fb2763f137059d754da2569
Nov 03 11:30:44.868700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:44.868203    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:45.500594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:45.495787    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:45.626930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:45.626893    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x" event={"ID":"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3","Type":"ContainerStarted","Data":"7fc587ef7c913ad775858bfa80564b1f5b7ce01fccdb0b0ae8d09b75c62d183a"}
Nov 03 11:30:45.627119 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:45.626979    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x" event={"ID":"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3","Type":"ContainerStarted","Data":"6110f4818b36c002b1bbbbb73998b30f0ae4e4464fb2763f137059d754da2569"}
Nov 03 11:30:45.644414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:45.644367    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x" podStartSLOduration=1.644353922 podStartE2EDuration="1.644353922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:45.642212163 +0000 UTC m=+3675.847794200" watchObservedRunningTime="2026-11-03 11:30:45.644353922 +0000 UTC m=+3675.849935972"
Nov 03 11:30:46.162198 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:46.161360    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:46.639126 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:46.639093    5817 generic.go:356] "Generic (PLEG): container finished" podID="1b130fdf-59ac-4f66-ad7c-2a04636e1dd3" containerID="7fc587ef7c913ad775858bfa80564b1f5b7ce01fccdb0b0ae8d09b75c62d183a" exitCode=0
Nov 03 11:30:46.639552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:46.639174    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x" event={"ID":"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3","Type":"ContainerDied","Data":"7fc587ef7c913ad775858bfa80564b1f5b7ce01fccdb0b0ae8d09b75c62d183a"}
Nov 03 11:30:46.861101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:46.860619    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:47.551756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.551723    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:47.808254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.808193    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:47.867715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.867686    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-xks-ca\") pod \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") "
Nov 03 11:30:47.867906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.867732    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-openshift-ca\") pod \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") "
Nov 03 11:30:47.867906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.867789    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-kube-api-access-6j47f\" (UniqueName: \"kubernetes.io/projected/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-kube-api-access-6j47f\") pod \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\" (UID: \"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3\") "
Nov 03 11:30:47.868176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.868148    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-openshift-ca" pod "1b130fdf-59ac-4f66-ad7c-2a04636e1dd3" (UID: "1b130fdf-59ac-4f66-ad7c-2a04636e1dd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:47.868468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.868413    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:47.870020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.869995    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-xks-ca" pod "1b130fdf-59ac-4f66-ad7c-2a04636e1dd3" (UID: "1b130fdf-59ac-4f66-ad7c-2a04636e1dd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:47.870205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.870177    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-kube-api-access-6j47f" pod "1b130fdf-59ac-4f66-ad7c-2a04636e1dd3" (UID: "1b130fdf-59ac-4f66-ad7c-2a04636e1dd3"). InnerVolumeSpecName "kube-api-access-6j47f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:47.969558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.969522    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j47f\" (UniqueName: \"kubernetes.io/projected/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-kube-api-access-6j47f\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:47.969750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:47.969564    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:48.054135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.054097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:48.133010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.132921    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"]
Nov 03 11:30:48.136342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.136313    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56687c-2vhgn"]
Nov 03 11:30:48.184567 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.184537    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:48.378596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.378459    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:48.713604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.708995    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:48.713604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.711946    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x" event={"ID":"1b130fdf-59ac-4f66-ad7c-2a04636e1dd3","Type":"ContainerDied","Data":"6110f4818b36c002b1bbbbb73998b30f0ae4e4464fb2763f137059d754da2569"}
Nov 03 11:30:48.713604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.712017    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6110f4818b36c002b1bbbbb73998b30f0ae4e4464fb2763f137059d754da2569"
Nov 03 11:30:48.713604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:48.712378    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"
Nov 03 11:30:49.224791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:49.224754    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:49.816596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:49.809161    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:49.910819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:49.910785    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ad61ab-2070-4932-923e-a167ed8c2ddf" path="/var/lib/kubelet/pods/56ad61ab-2070-4932-923e-a167ed8c2ddf/volumes"
Nov 03 11:30:50.479231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:50.479187    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:51.220675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:51.220628    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:51.863380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:51.863346    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:52.414088 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:30:52.414050    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:30:52.460111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:52.459673    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:53.121633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:53.120075    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:53.775911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:53.775874    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:54.448267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.448233    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"]
Nov 03 11:30:54.448885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.448870    5817 cpu_manager.go:414] "containerMap: removing container" podUID="1b130fdf-59ac-4f66-ad7c-2a04636e1dd3" containerName="revoke-keys"
Nov 03 11:30:54.448934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.448887    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b130fdf-59ac-4f66-ad7c-2a04636e1dd3" containerName="revoke-keys"
Nov 03 11:30:54.449003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.448993    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b130fdf-59ac-4f66-ad7c-2a04636e1dd3" containerName="revoke-keys"
Nov 03 11:30:54.453809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.453791    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.458787 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.458746    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:30:54.460168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.459960    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"]
Nov 03 11:30:54.460168 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.460042    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:30:54.474401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.474374    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdspg\" (UniqueName: \"kubernetes.io/projected/56b42064-852e-4039-940b-79c5bbfdc133-kube-api-access-rdspg\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.474548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.474525    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.474741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.474723    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.536446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.536349    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:54.576524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.576486    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.576777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.576629    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.576777 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.576744    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdspg\" (UniqueName: \"kubernetes.io/projected/56b42064-852e-4039-940b-79c5bbfdc133-kube-api-access-rdspg\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.577308 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.577287    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.579238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.579209    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.584973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.584949    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdspg\" (UniqueName: \"kubernetes.io/projected/56b42064-852e-4039-940b-79c5bbfdc133-kube-api-access-rdspg\") pod \"maas-api-revoke-sub-ff701a8c-p5lw8\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.772036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.771948    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:54.891728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:54.891691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:55.065625 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:30:55.065565    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b42064_852e_4039_940b_79c5bbfdc133.slice/crio-20b61fcf28f9c7963236982558e0b51821ebe7ac8086c4e68d94df9d86ce08a6 WatchSource:0}: Error finding container 20b61fcf28f9c7963236982558e0b51821ebe7ac8086c4e68d94df9d86ce08a6: Status 404 returned error can't find the container with id 20b61fcf28f9c7963236982558e0b51821ebe7ac8086c4e68d94df9d86ce08a6
Nov 03 11:30:55.066217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:55.066194    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"]
Nov 03 11:30:55.240724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:55.240691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:55.386486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:55.386399    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:55.596146 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:55.596114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:55.802019 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:55.801983    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8" event={"ID":"56b42064-852e-4039-940b-79c5bbfdc133","Type":"ContainerStarted","Data":"14d61053b1508e3f4d92303c034c7762938bc6baa9e93e246ddde712c2a65b26"}
Nov 03 11:30:55.802246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:55.802069    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8" event={"ID":"56b42064-852e-4039-940b-79c5bbfdc133","Type":"ContainerStarted","Data":"20b61fcf28f9c7963236982558e0b51821ebe7ac8086c4e68d94df9d86ce08a6"}
Nov 03 11:30:55.820627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:55.820564    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8" podStartSLOduration=1.820551419 podStartE2EDuration="1.820551419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:55.817695081 +0000 UTC m=+3686.023277145" watchObservedRunningTime="2026-11-03 11:30:55.820551419 +0000 UTC m=+3686.026133505"
Nov 03 11:30:55.945109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:55.945068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:56.815238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:56.815204    5817 generic.go:356] "Generic (PLEG): container finished" podID="56b42064-852e-4039-940b-79c5bbfdc133" containerID="14d61053b1508e3f4d92303c034c7762938bc6baa9e93e246ddde712c2a65b26" exitCode=0
Nov 03 11:30:56.815744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:56.815289    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8" event={"ID":"56b42064-852e-4039-940b-79c5bbfdc133","Type":"ContainerDied","Data":"14d61053b1508e3f4d92303c034c7762938bc6baa9e93e246ddde712c2a65b26"}
Nov 03 11:30:57.088111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:57.088016    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:57.586924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:57.578070    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:58.005510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.005479    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:58.131630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.131593    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b42064-852e-4039-940b-79c5bbfdc133-kube-api-access-rdspg\" (UniqueName: \"kubernetes.io/projected/56b42064-852e-4039-940b-79c5bbfdc133-kube-api-access-rdspg\") pod \"56b42064-852e-4039-940b-79c5bbfdc133\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") "
Nov 03 11:30:58.131879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.131767    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-openshift-ca\") pod \"56b42064-852e-4039-940b-79c5bbfdc133\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") "
Nov 03 11:30:58.131879 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.131833    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-xks-ca\") pod \"56b42064-852e-4039-940b-79c5bbfdc133\" (UID: \"56b42064-852e-4039-940b-79c5bbfdc133\") "
Nov 03 11:30:58.132820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.132731    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-openshift-ca" pod "56b42064-852e-4039-940b-79c5bbfdc133" (UID: "56b42064-852e-4039-940b-79c5bbfdc133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:58.135647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.134773    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b42064-852e-4039-940b-79c5bbfdc133-kube-api-access-rdspg" pod "56b42064-852e-4039-940b-79c5bbfdc133" (UID: "56b42064-852e-4039-940b-79c5bbfdc133"). InnerVolumeSpecName "kube-api-access-rdspg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:58.135647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.134851    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-xks-ca" pod "56b42064-852e-4039-940b-79c5bbfdc133" (UID: "56b42064-852e-4039-940b-79c5bbfdc133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:58.233773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.233739    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:58.233773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.233767    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b42064-852e-4039-940b-79c5bbfdc133-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:58.233773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.233779    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdspg\" (UniqueName: \"kubernetes.io/projected/56b42064-852e-4039-940b-79c5bbfdc133-kube-api-access-rdspg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:58.279726 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.279110    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:58.760522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.760490    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:30:58.837162 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.837130    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8"
Nov 03 11:30:58.837365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.837170    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff701a8c-p5lw8" event={"ID":"56b42064-852e-4039-940b-79c5bbfdc133","Type":"ContainerDied","Data":"20b61fcf28f9c7963236982558e0b51821ebe7ac8086c4e68d94df9d86ce08a6"}
Nov 03 11:30:58.837365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.837214    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b61fcf28f9c7963236982558e0b51821ebe7ac8086c4e68d94df9d86ce08a6"
Nov 03 11:30:58.894435 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:58.894390    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:30:59.872825 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:30:59.872789    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:00.884265 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:00.884222    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:01.518602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:01.513514    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:02.143998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:02.142247    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:02.752805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:02.752772    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:03.455601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:03.452560    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:04.024107 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:04.024074    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"]
Nov 03 11:31:04.029204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:04.029177    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f2a3b3e-82mk2"]
Nov 03 11:31:04.064934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:04.064898    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:04.765267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:04.765229    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:05.113929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:05.113850    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:05.469630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:05.467351    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:05.903636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:05.903538    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7e1d97-0679-4530-b888-fa0f86b5d3a4" path="/var/lib/kubelet/pods/4b7e1d97-0679-4530-b888-fa0f86b5d3a4/volumes"
Nov 03 11:31:06.166548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:06.166467    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"]
Nov 03 11:31:06.183988 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:06.183945    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"]
Nov 03 11:31:06.199375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:06.199344    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:06.347208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:06.347169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:06.967782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:06.967747    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:07.195935 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.195880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:07.315520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.315431    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"]
Nov 03 11:31:07.316133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.316115    5817 cpu_manager.go:414] "containerMap: removing container" podUID="56b42064-852e-4039-940b-79c5bbfdc133" containerName="revoke-keys"
Nov 03 11:31:07.316133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.316133    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b42064-852e-4039-940b-79c5bbfdc133" containerName="revoke-keys"
Nov 03 11:31:07.316279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.316223    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b42064-852e-4039-940b-79c5bbfdc133" containerName="revoke-keys"
Nov 03 11:31:07.320054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.320030    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.324293 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.324265    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:31:07.324411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.324332    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:31:07.329554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.329527    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"]
Nov 03 11:31:07.351071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.351043    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.351249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.351134    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.351391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.351365    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxdr\" (UniqueName: \"kubernetes.io/projected/9c78f579-e7ba-4df3-be80-f429d5d510d2-kube-api-access-hhxdr\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.452837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.452803    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxdr\" (UniqueName: \"kubernetes.io/projected/9c78f579-e7ba-4df3-be80-f429d5d510d2-kube-api-access-hhxdr\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.453018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.452885    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.453018 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.452925    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.453631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.453563    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.455434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.455411    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.463537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.463509    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxdr\" (UniqueName: \"kubernetes.io/projected/9c78f579-e7ba-4df3-be80-f429d5d510d2-kube-api-access-hhxdr\") pod \"maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.645383 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.645305    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:07.938626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.937743    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"]
Nov 03 11:31:07.958632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:07.958602    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf" event={"ID":"9c78f579-e7ba-4df3-be80-f429d5d510d2","Type":"ContainerStarted","Data":"15f0f535e0fed4d5eea277fbd99aeb8c8d6ee06d4250b81637c459ac36655c90"}
Nov 03 11:31:08.022597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:08.022051    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:08.906131 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:08.906099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:08.967522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:08.967491    5817 generic.go:356] "Generic (PLEG): container finished" podID="9c78f579-e7ba-4df3-be80-f429d5d510d2" containerID="55e814fac1fb82527037cae5b0ab1bab4980b268d212b528c045ed817e6c40c3" exitCode=0
Nov 03 11:31:08.967739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:08.967590    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf" event={"ID":"9c78f579-e7ba-4df3-be80-f429d5d510d2","Type":"ContainerDied","Data":"55e814fac1fb82527037cae5b0ab1bab4980b268d212b528c045ed817e6c40c3"}
Nov 03 11:31:09.159061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:09.158974    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:09.576607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:09.571708    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:10.215106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.215074    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:10.312872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.312784    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-xks-ca\") pod \"9c78f579-e7ba-4df3-be80-f429d5d510d2\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") "
Nov 03 11:31:10.312872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.312870    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-openshift-ca\") pod \"9c78f579-e7ba-4df3-be80-f429d5d510d2\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") "
Nov 03 11:31:10.313149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.312921    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c78f579-e7ba-4df3-be80-f429d5d510d2-kube-api-access-hhxdr\" (UniqueName: \"kubernetes.io/projected/9c78f579-e7ba-4df3-be80-f429d5d510d2-kube-api-access-hhxdr\") pod \"9c78f579-e7ba-4df3-be80-f429d5d510d2\" (UID: \"9c78f579-e7ba-4df3-be80-f429d5d510d2\") "
Nov 03 11:31:10.313284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.313259    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-openshift-ca" pod "9c78f579-e7ba-4df3-be80-f429d5d510d2" (UID: "9c78f579-e7ba-4df3-be80-f429d5d510d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:10.313398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.313381    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:10.315288 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.315264    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c78f579-e7ba-4df3-be80-f429d5d510d2-kube-api-access-hhxdr" pod "9c78f579-e7ba-4df3-be80-f429d5d510d2" (UID: "9c78f579-e7ba-4df3-be80-f429d5d510d2"). InnerVolumeSpecName "kube-api-access-hhxdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:10.315452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.315435    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-xks-ca" pod "9c78f579-e7ba-4df3-be80-f429d5d510d2" (UID: "9c78f579-e7ba-4df3-be80-f429d5d510d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:10.385992 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.385825    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:10.415016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.414979    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78f579-e7ba-4df3-be80-f429d5d510d2-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:10.415195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:10.415021    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhxdr\" (UniqueName: \"kubernetes.io/projected/9c78f579-e7ba-4df3-be80-f429d5d510d2-kube-api-access-hhxdr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:11.025098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:11.025070    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"
Nov 03 11:31:11.025286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:11.025076    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf" event={"ID":"9c78f579-e7ba-4df3-be80-f429d5d510d2","Type":"ContainerDied","Data":"15f0f535e0fed4d5eea277fbd99aeb8c8d6ee06d4250b81637c459ac36655c90"}
Nov 03 11:31:11.025286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:11.025191    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f0f535e0fed4d5eea277fbd99aeb8c8d6ee06d4250b81637c459ac36655c90"
Nov 03 11:31:11.167020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:11.166959    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l" podUID="4313de37-29db-4724-b33e-8783cfc1b979" containerName="payload-processing" containerID="cri-o://73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524" gracePeriod=25
Nov 03 11:31:11.184662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:11.184610    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq" podUID="654c883a-f036-4060-8989-2c4ee57dc590" containerName="payload-pre-processing" containerID="cri-o://219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387" gracePeriod=25
Nov 03 11:31:11.260363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:11.260324    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:11.985024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:11.984218    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:12.745559 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:12.745527    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:13.466552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:13.466449    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:14.154381 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:14.154188    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:14.761191 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:14.761156    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:15.240541 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:15.240504    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:15.420872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:15.420839    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:15.556967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:15.556887    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:15.694159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:15.694117    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:16.060445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:16.060412    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:16.316426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:16.316337    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l" podUID="4313de37-29db-4724-b33e-8783cfc1b979" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:16.316426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:16.316411    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq" podUID="654c883a-f036-4060-8989-2c4ee57dc590" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:16.367816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:16.366237    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:17.045679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:17.045592    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:17.439218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:17.439187    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"]
Nov 03 11:31:17.547758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:17.547725    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:17.686825 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:17.686786    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:17.846653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:17.846347    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:18.550285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:18.550166    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:18.550930 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:18.550909    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:31:18.671132 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:18.671104    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"]
Nov 03 11:31:18.675251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:18.675227    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-917da2-c5qtf"]
Nov 03 11:31:18.712621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:18.712561    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:18.855768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:18.855684    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:19.246242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:19.246204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:19.760346 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:19.760263    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:19.902540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:19.902507    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c78f579-e7ba-4df3-be80-f429d5d510d2" path="/var/lib/kubelet/pods/9c78f579-e7ba-4df3-be80-f429d5d510d2/volumes"
Nov 03 11:31:20.370249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:20.370214    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:21.067297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:21.067263    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:21.663714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:21.663675    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:22.338415 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:22.338375    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:22.412884 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:22.412848    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:31:22.440247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:22.440212    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r" podUID="f5fd6984-731f-43d9-98fc-0cfd8f2816fa" containerName="payload-pre-processing" containerID="cri-o://f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b" gracePeriod=25
Nov 03 11:31:22.928038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:22.928005    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:23.143616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.143583    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:23.270044 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.269957    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"]
Nov 03 11:31:23.270218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.270161    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy" containerID="cri-o://41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467" gracePeriod=30
Nov 03 11:31:23.414299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.414254    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:23.848758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.848728    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"]
Nov 03 11:31:23.849369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.849356    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9c78f579-e7ba-4df3-be80-f429d5d510d2" containerName="revoke-keys"
Nov 03 11:31:23.849417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.849371    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c78f579-e7ba-4df3-be80-f429d5d510d2" containerName="revoke-keys"
Nov 03 11:31:23.849472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.849463    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c78f579-e7ba-4df3-be80-f429d5d510d2" containerName="revoke-keys"
Nov 03 11:31:23.853654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.853634    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:23.894840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.893128    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:31:23.895213 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.895179    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:31:23.907274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.907245    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:23.914828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.914794    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"]
Nov 03 11:31:23.938742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.938714    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr84d\" (UniqueName: \"kubernetes.io/projected/aa1a64b7-c23b-4158-b47e-47df49f59d5f-kube-api-access-pr84d\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:23.938925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.938908    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:23.939206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:23.939184    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:24.040292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.040260    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:24.040473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.040308    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr84d\" (UniqueName: \"kubernetes.io/projected/aa1a64b7-c23b-4158-b47e-47df49f59d5f-kube-api-access-pr84d\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:24.040473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.040398    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:24.041069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.041050    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:24.042803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.042779    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:24.047925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.047900    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr84d\" (UniqueName: \"kubernetes.io/projected/aa1a64b7-c23b-4158-b47e-47df49f59d5f-kube-api-access-pr84d\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:24.127063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.126979    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:24.215937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.215911    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:24.510712 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:31:24.510684    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1a64b7_c23b_4158_b47e_47df49f59d5f.slice/crio-5452b2fef04ebb668dcbd22dfe6fe4f730c873ed7fd88fd3b6c7190bf5c4b874 WatchSource:0}: Error finding container 5452b2fef04ebb668dcbd22dfe6fe4f730c873ed7fd88fd3b6c7190bf5c4b874: Status 404 returned error can't find the container with id 5452b2fef04ebb668dcbd22dfe6fe4f730c873ed7fd88fd3b6c7190bf5c4b874
Nov 03 11:31:24.511065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.510748    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"]
Nov 03 11:31:24.854328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:24.854279    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:25.173619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:25.173496    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2" event={"ID":"aa1a64b7-c23b-4158-b47e-47df49f59d5f","Type":"ContainerStarted","Data":"10293c22e9c2218717246eec74d5c9750f54a81744ad73cc8c74dc1b41d18997"}
Nov 03 11:31:25.173619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:25.173553    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2" event={"ID":"aa1a64b7-c23b-4158-b47e-47df49f59d5f","Type":"ContainerStarted","Data":"5452b2fef04ebb668dcbd22dfe6fe4f730c873ed7fd88fd3b6c7190bf5c4b874"}
Nov 03 11:31:25.195080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:25.195023    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2" podStartSLOduration=2.195005808 podStartE2EDuration="2.195005808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:31:25.190747171 +0000 UTC m=+3715.396329223" watchObservedRunningTime="2026-11-03 11:31:25.195005808 +0000 UTC m=+3715.400587861"
Nov 03 11:31:25.485334 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:25.482309    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:26.080535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:26.080501    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:26.183193 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:26.183163    5817 generic.go:356] "Generic (PLEG): container finished" podID="aa1a64b7-c23b-4158-b47e-47df49f59d5f" containerID="10293c22e9c2218717246eec74d5c9750f54a81744ad73cc8c74dc1b41d18997" exitCode=0
Nov 03 11:31:26.183398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:26.183256    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2" event={"ID":"aa1a64b7-c23b-4158-b47e-47df49f59d5f","Type":"ContainerDied","Data":"10293c22e9c2218717246eec74d5c9750f54a81744ad73cc8c74dc1b41d18997"}
Nov 03 11:31:26.302278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:26.302233    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:26.320471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:26.320433    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq" podUID="654c883a-f036-4060-8989-2c4ee57dc590" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:26.320913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:26.320890    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l" podUID="4313de37-29db-4724-b33e-8783cfc1b979" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:26.461272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:26.461224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:26.811229 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:26.811102    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:27.363151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.363127    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:27.488667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.486481    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:27.519545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.519516    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-openshift-ca\") pod \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") "
Nov 03 11:31:27.519824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.519660    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1a64b7-c23b-4158-b47e-47df49f59d5f-kube-api-access-pr84d\" (UniqueName: \"kubernetes.io/projected/aa1a64b7-c23b-4158-b47e-47df49f59d5f-kube-api-access-pr84d\") pod \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") "
Nov 03 11:31:27.519824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.519788    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-xks-ca\") pod \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\" (UID: \"aa1a64b7-c23b-4158-b47e-47df49f59d5f\") "
Nov 03 11:31:27.519961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.519909    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-openshift-ca" pod "aa1a64b7-c23b-4158-b47e-47df49f59d5f" (UID: "aa1a64b7-c23b-4158-b47e-47df49f59d5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:27.520370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.520346    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:27.521967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.521944    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1a64b7-c23b-4158-b47e-47df49f59d5f-kube-api-access-pr84d" pod "aa1a64b7-c23b-4158-b47e-47df49f59d5f" (UID: "aa1a64b7-c23b-4158-b47e-47df49f59d5f"). InnerVolumeSpecName "kube-api-access-pr84d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:27.522135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.522109    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-xks-ca" pod "aa1a64b7-c23b-4158-b47e-47df49f59d5f" (UID: "aa1a64b7-c23b-4158-b47e-47df49f59d5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:27.621476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.621443    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr84d\" (UniqueName: \"kubernetes.io/projected/aa1a64b7-c23b-4158-b47e-47df49f59d5f-kube-api-access-pr84d\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:27.621476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.621468    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1a64b7-c23b-4158-b47e-47df49f59d5f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:27.784136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:27.784042    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r" podUID="f5fd6984-731f-43d9-98fc-0cfd8f2816fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:28.178393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.177919    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:28.215780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.215750    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"
Nov 03 11:31:28.215780 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.215774    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2" event={"ID":"aa1a64b7-c23b-4158-b47e-47df49f59d5f","Type":"ContainerDied","Data":"5452b2fef04ebb668dcbd22dfe6fe4f730c873ed7fd88fd3b6c7190bf5c4b874"}
Nov 03 11:31:28.216009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.215815    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5452b2fef04ebb668dcbd22dfe6fe4f730c873ed7fd88fd3b6c7190bf5c4b874"
Nov 03 11:31:28.463747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.463715    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 11:31:28.635102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635072    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-data\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635142    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-credential-socket\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635185    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-kube-api-access-65v8t\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-kube-api-access-65v8t\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635255    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istiod-ca-cert\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635299    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-token\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-token\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635332    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-socket\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635393    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-data" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.635564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635421    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-envoy\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635474    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-credential-socket" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.635564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635505    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-podinfo\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635708    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-certs\") pod \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\" (UID: \"695ee621-b3ee-4b7f-bc8a-bab4912f99a0\") "
Nov 03 11:31:28.635893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635806    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istiod-ca-cert" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:28.636027 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.635954    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-socket" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.636299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.636218    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-certs" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.636299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.636220    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.636299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.636269    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.636299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.636288    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.636636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.636302    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.638077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.638047    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-envoy" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.638242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.638220    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-podinfo" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:31:28.638734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.638703    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-kube-api-access-65v8t" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "kube-api-access-65v8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:28.638843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.638778    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-token" pod "695ee621-b3ee-4b7f-bc8a-bab4912f99a0" (UID: "695ee621-b3ee-4b7f-bc8a-bab4912f99a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:28.737626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.737541    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.737626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.737618    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65v8t\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-kube-api-access-65v8t\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.737848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.737635    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.737848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.737649    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.737848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:28.737663    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695ee621-b3ee-4b7f-bc8a-bab4912f99a0-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:29.047142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.047050    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:29.229279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.229245    5817 generic.go:356] "Generic (PLEG): container finished" podID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerID="41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467" exitCode=0
Nov 03 11:31:29.229548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.229285    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" event={"ID":"695ee621-b3ee-4b7f-bc8a-bab4912f99a0","Type":"ContainerDied","Data":"41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467"}
Nov 03 11:31:29.229548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.229297    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"
Nov 03 11:31:29.229548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.229338    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g" event={"ID":"695ee621-b3ee-4b7f-bc8a-bab4912f99a0","Type":"ContainerDied","Data":"09c3565fb8899804701d4072c55a7ce8cbbcb9a0e3c4e6c9708de384058c92d4"}
Nov 03 11:31:29.229548 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.229361    5817 scope.go:122] "RemoveContainer" containerID="41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467"
Nov 03 11:31:29.242052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.242028    5817 scope.go:122] "RemoveContainer" containerID="baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561"
Nov 03 11:31:29.252717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.252676    5817 scope.go:122] "RemoveContainer" containerID="41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467"
Nov 03 11:31:29.253073 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:29.253008    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467\": container with ID starting with 41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467 not found: ID does not exist" containerID="41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467"
Nov 03 11:31:29.253073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.253060    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467"} err="failed to get container status \"41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467\": rpc error: code = NotFound desc = could not find container \"41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467\": container with ID starting with 41839b96f82a564ced5bf4feadc00ff4c58ef4e94465b6353190916df1334467 not found: ID does not exist"
Nov 03 11:31:29.253304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.253086    5817 scope.go:122] "RemoveContainer" containerID="baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561"
Nov 03 11:31:29.253529 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:29.253466    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561\": container with ID starting with baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561 not found: ID does not exist" containerID="baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561"
Nov 03 11:31:29.253529 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.253500    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561"} err="failed to get container status \"baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561\": rpc error: code = NotFound desc = could not find container \"baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561\": container with ID starting with baa9d7cd3c6dfc0dd0301b33ee36afd38757c74c98bd36298e53b38fe1d6e561 not found: ID does not exist"
Nov 03 11:31:29.254371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.254349    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"]
Nov 03 11:31:29.258180 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.258160    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-m9q7g"]
Nov 03 11:31:29.680424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.679740    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:29.903661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:29.903631    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" path="/var/lib/kubelet/pods/695ee621-b3ee-4b7f-bc8a-bab4912f99a0/volumes"
Nov 03 11:31:30.341441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:30.341313    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:30.976382 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:30.976344    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:31.481177 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.481155    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 11:31:31.509870 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.509845    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 11:31:31.575379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.573849    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:31.582667 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.582631    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4313de37-29db-4724-b33e-8783cfc1b979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4313de37-29db-4724-b33e-8783cfc1b979-plugins-config-volume\") pod \"4313de37-29db-4724-b33e-8783cfc1b979\" (UID: \"4313de37-29db-4724-b33e-8783cfc1b979\") "
Nov 03 11:31:31.582878 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.582857    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654c883a-f036-4060-8989-2c4ee57dc590-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c883a-f036-4060-8989-2c4ee57dc590-plugins-config-volume\") pod \"654c883a-f036-4060-8989-2c4ee57dc590\" (UID: \"654c883a-f036-4060-8989-2c4ee57dc590\") "
Nov 03 11:31:31.583040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.582932    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4313de37-29db-4724-b33e-8783cfc1b979-kube-api-access-hpvdl\" (UniqueName: \"kubernetes.io/projected/4313de37-29db-4724-b33e-8783cfc1b979-kube-api-access-hpvdl\") pod \"4313de37-29db-4724-b33e-8783cfc1b979\" (UID: \"4313de37-29db-4724-b33e-8783cfc1b979\") "
Nov 03 11:31:31.583040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.582966    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654c883a-f036-4060-8989-2c4ee57dc590-kube-api-access-8gkl7\" (UniqueName: \"kubernetes.io/projected/654c883a-f036-4060-8989-2c4ee57dc590-kube-api-access-8gkl7\") pod \"654c883a-f036-4060-8989-2c4ee57dc590\" (UID: \"654c883a-f036-4060-8989-2c4ee57dc590\") "
Nov 03 11:31:31.583394 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.583261    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4313de37-29db-4724-b33e-8783cfc1b979-plugins-config-volume" pod "4313de37-29db-4724-b33e-8783cfc1b979" (UID: "4313de37-29db-4724-b33e-8783cfc1b979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:31.583718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.583689    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654c883a-f036-4060-8989-2c4ee57dc590-plugins-config-volume" pod "654c883a-f036-4060-8989-2c4ee57dc590" (UID: "654c883a-f036-4060-8989-2c4ee57dc590"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:31.585718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.585693    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4313de37-29db-4724-b33e-8783cfc1b979-kube-api-access-hpvdl" pod "4313de37-29db-4724-b33e-8783cfc1b979" (UID: "4313de37-29db-4724-b33e-8783cfc1b979"). InnerVolumeSpecName "kube-api-access-hpvdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:31.585940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.585914    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654c883a-f036-4060-8989-2c4ee57dc590-kube-api-access-8gkl7" pod "654c883a-f036-4060-8989-2c4ee57dc590" (UID: "654c883a-f036-4060-8989-2c4ee57dc590"). InnerVolumeSpecName "kube-api-access-8gkl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:31.684593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.684541    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gkl7\" (UniqueName: \"kubernetes.io/projected/654c883a-f036-4060-8989-2c4ee57dc590-kube-api-access-8gkl7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.684735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.684614    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4313de37-29db-4724-b33e-8783cfc1b979-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.684735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.684631    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c883a-f036-4060-8989-2c4ee57dc590-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.684735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:31.684646    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpvdl\" (UniqueName: \"kubernetes.io/projected/4313de37-29db-4724-b33e-8783cfc1b979-kube-api-access-hpvdl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:32.257432 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.257399    5817 generic.go:356] "Generic (PLEG): container finished" podID="4313de37-29db-4724-b33e-8783cfc1b979" containerID="73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524" exitCode=0
Nov 03 11:31:32.257916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.257461    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"
Nov 03 11:31:32.257916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.257491    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l" event={"ID":"4313de37-29db-4724-b33e-8783cfc1b979","Type":"ContainerDied","Data":"73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524"}
Nov 03 11:31:32.257916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.257553    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l" event={"ID":"4313de37-29db-4724-b33e-8783cfc1b979","Type":"ContainerDied","Data":"2fc95f5d5cce6fde9c2af635f76128e06af5aa74fe3f8361bdc239d52d953e97"}
Nov 03 11:31:32.257916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.257591    5817 scope.go:122] "RemoveContainer" containerID="73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524"
Nov 03 11:31:32.259379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.259352    5817 generic.go:356] "Generic (PLEG): container finished" podID="654c883a-f036-4060-8989-2c4ee57dc590" containerID="219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387" exitCode=0
Nov 03 11:31:32.259501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.259412    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"
Nov 03 11:31:32.259675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.259653    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq" event={"ID":"654c883a-f036-4060-8989-2c4ee57dc590","Type":"ContainerDied","Data":"219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387"}
Nov 03 11:31:32.259886 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.259865    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq" event={"ID":"654c883a-f036-4060-8989-2c4ee57dc590","Type":"ContainerDied","Data":"d92fe2f87fc82b758a37a107fd95fa8739846f5a908b89cc436f067ae52963bc"}
Nov 03 11:31:32.289842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.285467    5817 scope.go:122] "RemoveContainer" containerID="73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524"
Nov 03 11:31:32.289842 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:32.285961    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524\": container with ID starting with 73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524 not found: ID does not exist" containerID="73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524"
Nov 03 11:31:32.289842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.286000    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524"} err="failed to get container status \"73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524\": rpc error: code = NotFound desc = could not find container \"73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524\": container with ID starting with 73e7498538bad9871bf324fe97c9771c0e257ffc17dafcccc74927b14d405524 not found: ID does not exist"
Nov 03 11:31:32.289842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.286025    5817 scope.go:122] "RemoveContainer" containerID="219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387"
Nov 03 11:31:32.314419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.313386    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"]
Nov 03 11:31:32.316389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.315909    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:32.321362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.321338    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9c53b2-b89bff74f-snf7l"]
Nov 03 11:31:32.321514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.321493    5817 scope.go:122] "RemoveContainer" containerID="219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387"
Nov 03 11:31:32.321902 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:32.321876    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387\": container with ID starting with 219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387 not found: ID does not exist" containerID="219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387"
Nov 03 11:31:32.322026 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.321911    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387"} err="failed to get container status \"219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387\": rpc error: code = NotFound desc = could not find container \"219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387\": container with ID starting with 219a8a1d8e3003d874c4544c9c0591bf07005ac39c2c9fcb09c9a0d08e739387 not found: ID does not exist"
Nov 03 11:31:32.331547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.331520    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"]
Nov 03 11:31:32.334803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.334779    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9c53b2-56fb8d74bc-k82sq"]
Nov 03 11:31:32.895054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.895024    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"]
Nov 03 11:31:32.895260 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.895193    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy" containerID="cri-o://97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b" gracePeriod=30
Nov 03 11:31:32.956499 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:32.956464    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:33.677654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:33.676440    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:33.904103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:33.904064    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4313de37-29db-4724-b33e-8783cfc1b979" path="/var/lib/kubelet/pods/4313de37-29db-4724-b33e-8783cfc1b979/volumes"
Nov 03 11:31:33.904680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:33.904659    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654c883a-f036-4060-8989-2c4ee57dc590" path="/var/lib/kubelet/pods/654c883a-f036-4060-8989-2c4ee57dc590/volumes"
Nov 03 11:31:34.014526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.014494    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"]
Nov 03 11:31:34.031852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.031822    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"]
Nov 03 11:31:34.138073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.138034    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:34.334300 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.334166    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"]
Nov 03 11:31:34.335706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.335657    5817 cpu_manager.go:414] "containerMap: removing container" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy"
Nov 03 11:31:34.335894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.335879    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy"
Nov 03 11:31:34.336011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.335999    5817 cpu_manager.go:414] "containerMap: removing container" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy"
Nov 03 11:31:34.336100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.336090    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy"
Nov 03 11:31:34.336298 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.336280    5817 cpu_manager.go:414] "containerMap: removing container" podUID="4313de37-29db-4724-b33e-8783cfc1b979" containerName="payload-processing"
Nov 03 11:31:34.336444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.336430    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4313de37-29db-4724-b33e-8783cfc1b979" containerName="payload-processing"
Nov 03 11:31:34.336553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.336543    5817 cpu_manager.go:414] "containerMap: removing container" podUID="654c883a-f036-4060-8989-2c4ee57dc590" containerName="payload-pre-processing"
Nov 03 11:31:34.336696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.336683    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654c883a-f036-4060-8989-2c4ee57dc590" containerName="payload-pre-processing"
Nov 03 11:31:34.336782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.336773    5817 cpu_manager.go:414] "containerMap: removing container" podUID="aa1a64b7-c23b-4158-b47e-47df49f59d5f" containerName="revoke-keys"
Nov 03 11:31:34.336864 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.336855    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1a64b7-c23b-4158-b47e-47df49f59d5f" containerName="revoke-keys"
Nov 03 11:31:34.337120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.337100    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="654c883a-f036-4060-8989-2c4ee57dc590" containerName="payload-pre-processing"
Nov 03 11:31:34.337242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.337230    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1a64b7-c23b-4158-b47e-47df49f59d5f" containerName="revoke-keys"
Nov 03 11:31:34.340402 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.340376    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy"
Nov 03 11:31:34.340542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.340530    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="695ee621-b3ee-4b7f-bc8a-bab4912f99a0" containerName="istio-proxy"
Nov 03 11:31:34.340669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.340654    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="4313de37-29db-4724-b33e-8783cfc1b979" containerName="payload-processing"
Nov 03 11:31:34.346522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.345783    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"]
Nov 03 11:31:34.346522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.345903    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.349731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.337693    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:34.351206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.351187    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:31:34.351444 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.351427    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:31:34.491104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.491070    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:34.537631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.537595    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.537806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.537663    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfn4\" (UniqueName: \"kubernetes.io/projected/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-kube-api-access-qjfn4\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.537888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.537865    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.639317 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.639237    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.639491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.639327    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.639491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.639366    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfn4\" (UniqueName: \"kubernetes.io/projected/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-kube-api-access-qjfn4\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.640051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.640028    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.641849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.641827    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.648313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.648291    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfn4\" (UniqueName: \"kubernetes.io/projected/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-kube-api-access-qjfn4\") pod \"maas-api-revoke-sub-00426132-lsk55\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.689238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.689211    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:34.982644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:34.982613    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"]
Nov 03 11:31:34.991262 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:31:34.991194    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb812b1ad_4e6f_4e12_bdaf_4b1f70c2172a.slice/crio-303187d4f179a81d41bc2c1dc85284dd0495ee4d1f205ea10ec7dcd816601791 WatchSource:0}: Error finding container 303187d4f179a81d41bc2c1dc85284dd0495ee4d1f205ea10ec7dcd816601791: Status 404 returned error can't find the container with id 303187d4f179a81d41bc2c1dc85284dd0495ee4d1f205ea10ec7dcd816601791
Nov 03 11:31:35.070165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:35.067629    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:35.300414 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:35.300376    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:35.327408 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:35.327369    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55" event={"ID":"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a","Type":"ContainerStarted","Data":"54bab49f84365ede2dfbea99292393b4a4f0978c9eab330cb925472876e86565"}
Nov 03 11:31:35.327526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:35.327435    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55" event={"ID":"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a","Type":"ContainerStarted","Data":"303187d4f179a81d41bc2c1dc85284dd0495ee4d1f205ea10ec7dcd816601791"}
Nov 03 11:31:35.342904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:35.342856    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55" podStartSLOduration=1.342839703 podStartE2EDuration="1.342839703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:31:35.342359425 +0000 UTC m=+3725.547941477" watchObservedRunningTime="2026-11-03 11:31:35.342839703 +0000 UTC m=+3725.548421756"
Nov 03 11:31:35.435286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:35.435253    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:35.756647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:35.755955    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:36.046101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:36.046023    5817 scope.go:122] "RemoveContainer" containerID="d086071bb6974cf5033d9ded3c29643d55b044ba2f92402a936d794a3dfec2fd"
Nov 03 11:31:36.338400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:36.338313    5817 generic.go:356] "Generic (PLEG): container finished" podID="b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a" containerID="54bab49f84365ede2dfbea99292393b4a4f0978c9eab330cb925472876e86565" exitCode=0
Nov 03 11:31:36.338587 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:36.338395    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55" event={"ID":"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a","Type":"ContainerDied","Data":"54bab49f84365ede2dfbea99292393b4a4f0978c9eab330cb925472876e86565"}
Nov 03 11:31:36.454599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:36.454391    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:36.516172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:36.516140    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:31:37.220142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.220108    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"]
Nov 03 11:31:37.225821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.225801    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.231525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.231502    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-836589-openshift-default-dockercfg-hw772\""
Nov 03 11:31:37.237322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.237295    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"]
Nov 03 11:31:37.299654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.299623    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.299830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.299658    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-podinfo\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.299830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.299685    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-certs\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.299830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.299780    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-credential-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.299830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.299819    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-envoy\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.300349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.299897    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlc7m\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-kube-api-access-vlc7m\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.300349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.300002    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-token\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.300349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.300019    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f820a885-2323-41ca-aaa5-0f55ccd76134-istiod-ca-cert\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.300349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.300068    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-data\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.401810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.401775    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-token\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.401815    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f820a885-2323-41ca-aaa5-0f55ccd76134-istiod-ca-cert\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.401872    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-data\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.401940    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.401966    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-podinfo\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.401997    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-certs\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402319 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.402045    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-credential-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402319 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.402107    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-envoy\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402319 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.402173    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlc7m\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-kube-api-access-vlc7m\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.402816    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f820a885-2323-41ca-aaa5-0f55ccd76134-istiod-ca-cert\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.402994 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.402824    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-certs\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.403064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.403005    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-data\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.403167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.403141    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.403304 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.403275    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-credential-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.405094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.405035    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-podinfo\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.416649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.414987    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlc7m\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-kube-api-access-vlc7m\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.417505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.417482    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-token\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.424036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.423951    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-envoy\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.513165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.513138    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:37.544238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.544209    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:37.607287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.604696    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-xks-ca\") pod \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") "
Nov 03 11:31:37.607287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.605413    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-kube-api-access-qjfn4\" (UniqueName: \"kubernetes.io/projected/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-kube-api-access-qjfn4\") pod \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") "
Nov 03 11:31:37.607287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.605520    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-openshift-ca\") pod \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\" (UID: \"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a\") "
Nov 03 11:31:37.607287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.606490    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-openshift-ca" pod "b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a" (UID: "b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:37.607749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.607311    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-xks-ca" pod "b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a" (UID: "b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:37.609429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.609382    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-kube-api-access-qjfn4" pod "b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a" (UID: "b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a"). InnerVolumeSpecName "kube-api-access-qjfn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:37.678189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.678153    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:37.707647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.707611    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjfn4\" (UniqueName: \"kubernetes.io/projected/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-kube-api-access-qjfn4\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:37.707647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.707645    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:37.707859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.707660    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:37.784258 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.784167    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r" podUID="f5fd6984-731f-43d9-98fc-0cfd8f2816fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:37.849723 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:31:37.849695    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf820a885_2323_41ca_aaa5_0f55ccd76134.slice/crio-7292e463efbbaf2157f8a3f703ae10d9d00fe4122cf7c5e57ca31ac3ac00ec42 WatchSource:0}: Error finding container 7292e463efbbaf2157f8a3f703ae10d9d00fe4122cf7c5e57ca31ac3ac00ec42: Status 404 returned error can't find the container with id 7292e463efbbaf2157f8a3f703ae10d9d00fe4122cf7c5e57ca31ac3ac00ec42
Nov 03 11:31:37.850369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.850343    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"]
Nov 03 11:31:37.852172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.852138    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:31:37.852267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.852207    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:31:37.852267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:37.852236    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:31:38.158747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.158722    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.217972    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-data\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218062    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07128b9c-8528-46df-9b95-7de2110c2b6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07128b9c-8528-46df-9b95-7de2110c2b6b-istiod-ca-cert\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218094    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-credential-socket\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218114    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-podinfo\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218158    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-token\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-token\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218189    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-socket\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218228    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-kube-api-access-pnbpz\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-kube-api-access-pnbpz\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218363    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-envoy\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218415    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-certs\") pod \"07128b9c-8528-46df-9b95-7de2110c2b6b\" (UID: \"07128b9c-8528-46df-9b95-7de2110c2b6b\") "
Nov 03 11:31:38.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218597    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-credential-socket" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:38.219540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218964    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07128b9c-8528-46df-9b95-7de2110c2b6b-istiod-ca-cert" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:38.219540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.218984    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.219540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.219214    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-socket" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:38.219540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.219264    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-certs" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:38.219540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.219463    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-data" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:38.222403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.222361    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-envoy" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:38.222403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.222383    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-podinfo" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:31:38.223048 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.222533    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-kube-api-access-pnbpz" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "kube-api-access-pnbpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:38.223854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.223828    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-token" pod "07128b9c-8528-46df-9b95-7de2110c2b6b" (UID: "07128b9c-8528-46df-9b95-7de2110c2b6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:38.320685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.320646    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.320685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.320676    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07128b9c-8528-46df-9b95-7de2110c2b6b-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.320685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.320686    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.320685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.320695    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.321106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.320704    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.321106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.320713    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnbpz\" (UniqueName: \"kubernetes.io/projected/07128b9c-8528-46df-9b95-7de2110c2b6b-kube-api-access-pnbpz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.321106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.320721    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.321106 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.320730    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07128b9c-8528-46df-9b95-7de2110c2b6b-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.329026 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.328939    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"]
Nov 03 11:31:38.329227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.329201    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4" podUID="6a1a5397-8f6b-4415-a277-53804bb59b46" containerName="discovery" containerID="cri-o://1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048" gracePeriod=30
Nov 03 11:31:38.364526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.364499    5817 generic.go:356] "Generic (PLEG): container finished" podID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerID="97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b" exitCode=0
Nov 03 11:31:38.364670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.364558    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"
Nov 03 11:31:38.364670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.364595    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" event={"ID":"07128b9c-8528-46df-9b95-7de2110c2b6b","Type":"ContainerDied","Data":"97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b"}
Nov 03 11:31:38.364670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.364657    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5" event={"ID":"07128b9c-8528-46df-9b95-7de2110c2b6b","Type":"ContainerDied","Data":"63328049468ae0d6fc947b50ae7b4102066dafdcb250a0642095039fae460867"}
Nov 03 11:31:38.364837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.364689    5817 scope.go:122] "RemoveContainer" containerID="97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b"
Nov 03 11:31:38.366696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.366547    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" event={"ID":"f820a885-2323-41ca-aaa5-0f55ccd76134","Type":"ContainerStarted","Data":"e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9"}
Nov 03 11:31:38.366696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.366612    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" event={"ID":"f820a885-2323-41ca-aaa5-0f55ccd76134","Type":"ContainerStarted","Data":"7292e463efbbaf2157f8a3f703ae10d9d00fe4122cf7c5e57ca31ac3ac00ec42"}
Nov 03 11:31:38.368157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.368136    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55" event={"ID":"b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a","Type":"ContainerDied","Data":"303187d4f179a81d41bc2c1dc85284dd0495ee4d1f205ea10ec7dcd816601791"}
Nov 03 11:31:38.368267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.368176    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="303187d4f179a81d41bc2c1dc85284dd0495ee4d1f205ea10ec7dcd816601791"
Nov 03 11:31:38.368267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.368231    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00426132-lsk55"
Nov 03 11:31:38.396080 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.396030    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podStartSLOduration=1.396013635 podStartE2EDuration="1.396013635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:31:38.389701234 +0000 UTC m=+3728.595283474" watchObservedRunningTime="2026-11-03 11:31:38.396013635 +0000 UTC m=+3728.601595684"
Nov 03 11:31:38.455362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.455094    5817 scope.go:122] "RemoveContainer" containerID="97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b"
Nov 03 11:31:38.455663 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:38.455628    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b\": container with ID starting with 97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b not found: ID does not exist" containerID="97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b"
Nov 03 11:31:38.455781 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.455673    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b"} err="failed to get container status \"97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b\": rpc error: code = NotFound desc = could not find container \"97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b\": container with ID starting with 97ae93be445399e2a5d28a4d424bf420c86f63a941ee8517e84ecd3be2a5980b not found: ID does not exist"
Nov 03 11:31:38.455955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.455936    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"]
Nov 03 11:31:38.470592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.462642    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-x6tv5"]
Nov 03 11:31:38.485590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.485454    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:38.544796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.544769    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:38.645745 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.645722    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 11:31:38.725481 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.725453    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-ca-configmap\") pod \"6a1a5397-8f6b-4415-a277-53804bb59b46\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") "
Nov 03 11:31:38.725685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.725521    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-kube-api-access-k64dq\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-kube-api-access-k64dq\") pod \"6a1a5397-8f6b-4415-a277-53804bb59b46\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") "
Nov 03 11:31:38.725685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.725628    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-cacerts\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-cacerts\") pod \"6a1a5397-8f6b-4415-a277-53804bb59b46\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") "
Nov 03 11:31:38.725685 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.725668    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-token\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-token\") pod \"6a1a5397-8f6b-4415-a277-53804bb59b46\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") "
Nov 03 11:31:38.725901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.725702    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1a5397-8f6b-4415-a277-53804bb59b46-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1a5397-8f6b-4415-a277-53804bb59b46-local-certs\") pod \"6a1a5397-8f6b-4415-a277-53804bb59b46\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") "
Nov 03 11:31:38.725901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.725760    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-kubeconfig\") pod \"6a1a5397-8f6b-4415-a277-53804bb59b46\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") "
Nov 03 11:31:38.725901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.725851    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-dns-cert\") pod \"6a1a5397-8f6b-4415-a277-53804bb59b46\" (UID: \"6a1a5397-8f6b-4415-a277-53804bb59b46\") "
Nov 03 11:31:38.726100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.725896    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-ca-configmap" pod "6a1a5397-8f6b-4415-a277-53804bb59b46" (UID: "6a1a5397-8f6b-4415-a277-53804bb59b46"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:38.726597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.726550    5817 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-ca-configmap\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.729009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.728983    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-kube-api-access-k64dq" pod "6a1a5397-8f6b-4415-a277-53804bb59b46" (UID: "6a1a5397-8f6b-4415-a277-53804bb59b46"). InnerVolumeSpecName "kube-api-access-k64dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:38.729155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.728986    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-cacerts" pod "6a1a5397-8f6b-4415-a277-53804bb59b46" (UID: "6a1a5397-8f6b-4415-a277-53804bb59b46"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:38.729155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.729031    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-kubeconfig" pod "6a1a5397-8f6b-4415-a277-53804bb59b46" (UID: "6a1a5397-8f6b-4415-a277-53804bb59b46"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:38.729406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.729390    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-token" pod "6a1a5397-8f6b-4415-a277-53804bb59b46" (UID: "6a1a5397-8f6b-4415-a277-53804bb59b46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:38.729520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.729483    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1a5397-8f6b-4415-a277-53804bb59b46-local-certs" pod "6a1a5397-8f6b-4415-a277-53804bb59b46" (UID: "6a1a5397-8f6b-4415-a277-53804bb59b46"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:38.730231 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.730206    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-dns-cert" pod "6a1a5397-8f6b-4415-a277-53804bb59b46" (UID: "6a1a5397-8f6b-4415-a277-53804bb59b46"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:38.827973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.827928    5817 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-cacerts\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.827973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.827974    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.828188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.827991    5817 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1a5397-8f6b-4415-a277-53804bb59b46-local-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.828188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.828004    5817 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-kubeconfig\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.828188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.828021    5817 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a1a5397-8f6b-4415-a277-53804bb59b46-istio-csr-dns-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:38.828188 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:38.828036    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k64dq\" (UniqueName: \"kubernetes.io/projected/6a1a5397-8f6b-4415-a277-53804bb59b46-kube-api-access-k64dq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:39.018756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.016647    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh" podUID="dd560ff4-8e15-492e-9278-c636496dd2be" containerName="payload-processing" containerID="cri-o://27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe" gracePeriod=25
Nov 03 11:31:39.033224 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.033185    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt" podUID="c6737167-e4c9-4287-bd9e-9151f1f14eed" containerName="payload-pre-processing" containerID="cri-o://34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470" gracePeriod=25
Nov 03 11:31:39.043899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.043867    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:39.182876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.182839    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:39.218680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.218635    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"]
Nov 03 11:31:39.226463 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.226419    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ef1662-2k4n2"]
Nov 03 11:31:39.319513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.319340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:39.383825 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.383791    5817 generic.go:356] "Generic (PLEG): container finished" podID="6a1a5397-8f6b-4415-a277-53804bb59b46" containerID="1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048" exitCode=0
Nov 03 11:31:39.384124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.383841    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"
Nov 03 11:31:39.402609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.383859    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4" event={"ID":"6a1a5397-8f6b-4415-a277-53804bb59b46","Type":"ContainerDied","Data":"1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048"}
Nov 03 11:31:39.402609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.384810    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4" event={"ID":"6a1a5397-8f6b-4415-a277-53804bb59b46","Type":"ContainerDied","Data":"57712cc7d8a329a3637e569a0fdd1fb64dd27b4b5a29a5fc5269418a2c8ccafe"}
Nov 03 11:31:39.402609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.384843    5817 scope.go:122] "RemoveContainer" containerID="1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048"
Nov 03 11:31:39.450926 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.450898    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"]
Nov 03 11:31:39.452616 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.452594    5817 scope.go:122] "RemoveContainer" containerID="1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048"
Nov 03 11:31:39.453233 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:39.453213    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048\": container with ID starting with 1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048 not found: ID does not exist" containerID="1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048"
Nov 03 11:31:39.453360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.453341    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048"} err="failed to get container status \"1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048\": rpc error: code = NotFound desc = could not find container \"1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048\": container with ID starting with 1b9872eff930086fb08d571a2c01926267e03cf2d88337433211fa0676015048 not found: ID does not exist"
Nov 03 11:31:39.456248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.456226    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8jg4"]
Nov 03 11:31:39.464691 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.464533    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:39.546867 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.546837    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused" start-of-body=
Nov 03 11:31:39.547041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.546889    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused"
Nov 03 11:31:39.736154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.736123    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:39.924090 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.924058    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" path="/var/lib/kubelet/pods/07128b9c-8528-46df-9b95-7de2110c2b6b/volumes"
Nov 03 11:31:39.924607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.924588    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1a5397-8f6b-4415-a277-53804bb59b46" path="/var/lib/kubelet/pods/6a1a5397-8f6b-4415-a277-53804bb59b46/volumes"
Nov 03 11:31:39.925020 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:39.925007    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1a64b7-c23b-4158-b47e-47df49f59d5f" path="/var/lib/kubelet/pods/aa1a64b7-c23b-4158-b47e-47df49f59d5f/volumes"
Nov 03 11:31:40.174630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:40.174407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:40.545336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:40.545307    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused" start-of-body=
Nov 03 11:31:40.545758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:40.545357    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused"
Nov 03 11:31:40.863520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:40.863433    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:41.474593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:41.472179    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:41.545089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:41.545059    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused" start-of-body=
Nov 03 11:31:41.545271 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:41.545114    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused"
Nov 03 11:31:42.177040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.177001    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:42.545104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.545066    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused" start-of-body=
Nov 03 11:31:42.545291 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.545124    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused"
Nov 03 11:31:42.773666 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.773643    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:31:42.871612 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.870033    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:42.885785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.885757    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-plugins-config-volume\") pod \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\" (UID: \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\") "
Nov 03 11:31:42.886013 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.885996    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-kube-api-access-xs4rj\" (UniqueName: \"kubernetes.io/projected/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-kube-api-access-xs4rj\") pod \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\" (UID: \"f5fd6984-731f-43d9-98fc-0cfd8f2816fa\") "
Nov 03 11:31:42.886167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.886141    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-plugins-config-volume" pod "f5fd6984-731f-43d9-98fc-0cfd8f2816fa" (UID: "f5fd6984-731f-43d9-98fc-0cfd8f2816fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:42.886433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.886416    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:42.918661 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.918625    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-kube-api-access-xs4rj" pod "f5fd6984-731f-43d9-98fc-0cfd8f2816fa" (UID: "f5fd6984-731f-43d9-98fc-0cfd8f2816fa"). InnerVolumeSpecName "kube-api-access-xs4rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:42.990595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:42.987293    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs4rj\" (UniqueName: \"kubernetes.io/projected/f5fd6984-731f-43d9-98fc-0cfd8f2816fa-kube-api-access-xs4rj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:43.446512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.446478    5817 generic.go:356] "Generic (PLEG): container finished" podID="f5fd6984-731f-43d9-98fc-0cfd8f2816fa" containerID="f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b" exitCode=0
Nov 03 11:31:43.447045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.446529    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"
Nov 03 11:31:43.447045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.446564    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r" event={"ID":"f5fd6984-731f-43d9-98fc-0cfd8f2816fa","Type":"ContainerDied","Data":"f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b"}
Nov 03 11:31:43.447045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.446646    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r" event={"ID":"f5fd6984-731f-43d9-98fc-0cfd8f2816fa","Type":"ContainerDied","Data":"e78f5a0a9204fe41f79c01fead88bac8365bac503e908d74766ad202af7c2a4b"}
Nov 03 11:31:43.447045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.446671    5817 scope.go:122] "RemoveContainer" containerID="f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b"
Nov 03 11:31:43.458037 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.458018    5817 scope.go:122] "RemoveContainer" containerID="f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b"
Nov 03 11:31:43.458340 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:43.458324    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b\": container with ID starting with f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b not found: ID does not exist" containerID="f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b"
Nov 03 11:31:43.458404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.458351    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b"} err="failed to get container status \"f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b\": rpc error: code = NotFound desc = could not find container \"f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b\": container with ID starting with f6730a671933681a07ca8fb7295740d78a43e3cfccdc5e5ebc2ae0a248506f5b not found: ID does not exist"
Nov 03 11:31:43.474277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.474254    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"]
Nov 03 11:31:43.486108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.486033    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-917da2-5ddb545f6c-8942r"]
Nov 03 11:31:43.544900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.544870    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused" start-of-body=
Nov 03 11:31:43.545065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.544926    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused"
Nov 03 11:31:43.578324 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.577879    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:43.924620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:43.924517    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fd6984-731f-43d9-98fc-0cfd8f2816fa" path="/var/lib/kubelet/pods/f5fd6984-731f-43d9-98fc-0cfd8f2816fa/volumes"
Nov 03 11:31:44.173956 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.173786    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:44.545083 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.545051    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused" start-of-body=
Nov 03 11:31:44.545506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.545105    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.138:15021/healthz/ready\": dial tcp 10.133.1.138:15021: connect: connection refused"
Nov 03 11:31:44.863795 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.863692    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:44.964851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.964818    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"]
Nov 03 11:31:44.965373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965360    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a" containerName="revoke-keys"
Nov 03 11:31:44.965422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965374    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a" containerName="revoke-keys"
Nov 03 11:31:44.965422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965409    5817 cpu_manager.go:414] "containerMap: removing container" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy"
Nov 03 11:31:44.965422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965414    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy"
Nov 03 11:31:44.965528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965432    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f5fd6984-731f-43d9-98fc-0cfd8f2816fa" containerName="payload-pre-processing"
Nov 03 11:31:44.965528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965438    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fd6984-731f-43d9-98fc-0cfd8f2816fa" containerName="payload-pre-processing"
Nov 03 11:31:44.965528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965447    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6a1a5397-8f6b-4415-a277-53804bb59b46" containerName="discovery"
Nov 03 11:31:44.965528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965452    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1a5397-8f6b-4415-a277-53804bb59b46" containerName="discovery"
Nov 03 11:31:44.965528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965527    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fd6984-731f-43d9-98fc-0cfd8f2816fa" containerName="payload-pre-processing"
Nov 03 11:31:44.965719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965535    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1a5397-8f6b-4415-a277-53804bb59b46" containerName="discovery"
Nov 03 11:31:44.965719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965545    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a" containerName="revoke-keys"
Nov 03 11:31:44.965719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.965555    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="07128b9c-8528-46df-9b95-7de2110c2b6b" containerName="istio-proxy"
Nov 03 11:31:44.970521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.970500    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:44.975157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.975030    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:31:44.975157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.975129    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:31:44.976683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:44.976664    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"]
Nov 03 11:31:45.037819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.037784    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntbt\" (UniqueName: \"kubernetes.io/projected/f043817c-c3d3-4a48-a8eb-de7a02663aad-kube-api-access-jntbt\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.038152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.038118    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.038337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.038306    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.140038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.139954    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntbt\" (UniqueName: \"kubernetes.io/projected/f043817c-c3d3-4a48-a8eb-de7a02663aad-kube-api-access-jntbt\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.140038 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.140009    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.140201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.140096    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.140825 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.140808    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.142465 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.142441    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.147823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.147801    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntbt\" (UniqueName: \"kubernetes.io/projected/f043817c-c3d3-4a48-a8eb-de7a02663aad-kube-api-access-jntbt\") pod \"maas-api-revoke-sub-fd6e98d4-svtns\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.302286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.302255    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:45.550011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.549981    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:45.550411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.550380    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:45.551259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.551239    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:31:45.586338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.586309    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"]
Nov 03 11:31:45.586887 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:31:45.586860    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf043817c_c3d3_4a48_a8eb_de7a02663aad.slice/crio-a1dc7c6e30e446064d38c5b41f11c2f16c82e94d8164edee8323b2459b682520 WatchSource:0}: Error finding container a1dc7c6e30e446064d38c5b41f11c2f16c82e94d8164edee8323b2459b682520: Status 404 returned error can't find the container with id a1dc7c6e30e446064d38c5b41f11c2f16c82e94d8164edee8323b2459b682520
Nov 03 11:31:45.616272 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:45.616241    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:46.175433 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.175351    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:46.318532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.318462    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:46.431283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.431256    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"]
Nov 03 11:31:46.431555 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.431508    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" containerID="cri-o://9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04" gracePeriod=30
Nov 03 11:31:46.476792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.476756    5817 generic.go:356] "Generic (PLEG): container finished" podID="f043817c-c3d3-4a48-a8eb-de7a02663aad" containerID="139cd0dd2ccf71a2e1b1b5b88d600375513b1b4a2e1fe582aadddbe6910ff9d5" exitCode=0
Nov 03 11:31:46.477022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.476832    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns" event={"ID":"f043817c-c3d3-4a48-a8eb-de7a02663aad","Type":"ContainerDied","Data":"139cd0dd2ccf71a2e1b1b5b88d600375513b1b4a2e1fe582aadddbe6910ff9d5"}
Nov 03 11:31:46.477022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.476894    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns" event={"ID":"f043817c-c3d3-4a48-a8eb-de7a02663aad","Type":"ContainerStarted","Data":"a1dc7c6e30e446064d38c5b41f11c2f16c82e94d8164edee8323b2459b682520"}
Nov 03 11:31:46.522785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.522756    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:46.794380 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.794293    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt" podUID="c6737167-e4c9-4287-bd9e-9151f1f14eed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:46.794869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.794714    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh" podUID="dd560ff4-8e15-492e-9278-c636496dd2be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:46.994150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:46.994118    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:47.121054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.120959    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"]
Nov 03 11:31:47.125443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.125422    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.133285 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.133263    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"]
Nov 03 11:31:47.144702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.144664    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:47.299595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.298361    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.299595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.298664    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvnh\" (UniqueName: \"kubernetes.io/projected/695046b6-8796-44af-bf28-0861bbfa0497-kube-api-access-qrvnh\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.299595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.298706    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.400565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.400474    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvnh\" (UniqueName: \"kubernetes.io/projected/695046b6-8796-44af-bf28-0861bbfa0497-kube-api-access-qrvnh\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.400565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.400513    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.400565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.400560    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.401427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.401391    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.403369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.403336    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.409141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.409112    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvnh\" (UniqueName: \"kubernetes.io/projected/695046b6-8796-44af-bf28-0861bbfa0497-kube-api-access-qrvnh\") pod \"maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.444153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.444127    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:47.729657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.729629    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:47.737945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.737925    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"]
Nov 03 11:31:47.738802 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:31:47.738773    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695046b6_8796_44af_bf28_0861bbfa0497.slice/crio-01ef956141b1d46356e87a349013ca389017745b0a36a3d1be10a54e417ecedd WatchSource:0}: Error finding container 01ef956141b1d46356e87a349013ca389017745b0a36a3d1be10a54e417ecedd: Status 404 returned error can't find the container with id 01ef956141b1d46356e87a349013ca389017745b0a36a3d1be10a54e417ecedd
Nov 03 11:31:47.740753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.740722    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:47.908950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.908916    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-xks-ca\") pod \"f043817c-c3d3-4a48-a8eb-de7a02663aad\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") "
Nov 03 11:31:47.909406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.909015    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f043817c-c3d3-4a48-a8eb-de7a02663aad-kube-api-access-jntbt\" (UniqueName: \"kubernetes.io/projected/f043817c-c3d3-4a48-a8eb-de7a02663aad-kube-api-access-jntbt\") pod \"f043817c-c3d3-4a48-a8eb-de7a02663aad\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") "
Nov 03 11:31:47.909406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.909053    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-openshift-ca\") pod \"f043817c-c3d3-4a48-a8eb-de7a02663aad\" (UID: \"f043817c-c3d3-4a48-a8eb-de7a02663aad\") "
Nov 03 11:31:47.909862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.909835    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-openshift-ca" pod "f043817c-c3d3-4a48-a8eb-de7a02663aad" (UID: "f043817c-c3d3-4a48-a8eb-de7a02663aad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:47.914025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.913601    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f043817c-c3d3-4a48-a8eb-de7a02663aad-kube-api-access-jntbt" pod "f043817c-c3d3-4a48-a8eb-de7a02663aad" (UID: "f043817c-c3d3-4a48-a8eb-de7a02663aad"). InnerVolumeSpecName "kube-api-access-jntbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:47.916214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:47.916188    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-xks-ca" pod "f043817c-c3d3-4a48-a8eb-de7a02663aad" (UID: "f043817c-c3d3-4a48-a8eb-de7a02663aad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:48.010932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.010843    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jntbt\" (UniqueName: \"kubernetes.io/projected/f043817c-c3d3-4a48-a8eb-de7a02663aad-kube-api-access-jntbt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:48.010932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.010877    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:48.010932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.010893    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043817c-c3d3-4a48-a8eb-de7a02663aad-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:48.115226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.115195    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:48.332305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.332205    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:48.497475 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.497444    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q" event={"ID":"695046b6-8796-44af-bf28-0861bbfa0497","Type":"ContainerStarted","Data":"2523b87a3f5a89c5c93e416e15e4678493bba307c2981c5a9934a0db03f91383"}
Nov 03 11:31:48.497683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.497502    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q" event={"ID":"695046b6-8796-44af-bf28-0861bbfa0497","Type":"ContainerStarted","Data":"01ef956141b1d46356e87a349013ca389017745b0a36a3d1be10a54e417ecedd"}
Nov 03 11:31:48.499208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.499176    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns"
Nov 03 11:31:48.499336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.499214    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6e98d4-svtns" event={"ID":"f043817c-c3d3-4a48-a8eb-de7a02663aad","Type":"ContainerDied","Data":"a1dc7c6e30e446064d38c5b41f11c2f16c82e94d8164edee8323b2459b682520"}
Nov 03 11:31:48.499336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.499260    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1dc7c6e30e446064d38c5b41f11c2f16c82e94d8164edee8323b2459b682520"
Nov 03 11:31:48.516139 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.516098    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q" podStartSLOduration=1.516086263 podStartE2EDuration="1.516086263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:31:48.512532739 +0000 UTC m=+3738.718114794" watchObservedRunningTime="2026-11-03 11:31:48.516086263 +0000 UTC m=+3738.721668313"
Nov 03 11:31:48.786759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:48.786719    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:49.158316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:49.157999    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:49.511920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:49.511890    5817 generic.go:356] "Generic (PLEG): container finished" podID="695046b6-8796-44af-bf28-0861bbfa0497" containerID="2523b87a3f5a89c5c93e416e15e4678493bba307c2981c5a9934a0db03f91383" exitCode=0
Nov 03 11:31:49.512113 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:49.511984    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q" event={"ID":"695046b6-8796-44af-bf28-0861bbfa0497","Type":"ContainerDied","Data":"2523b87a3f5a89c5c93e416e15e4678493bba307c2981c5a9934a0db03f91383"}
Nov 03 11:31:49.577297 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:49.577261    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:31:49.577517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:49.577316    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:31:49.854246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:49.854155    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:50.455554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.455517    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:50.676186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.676163    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:50.847166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.847075    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-xks-ca\") pod \"695046b6-8796-44af-bf28-0861bbfa0497\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") "
Nov 03 11:31:50.847332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.847215    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695046b6-8796-44af-bf28-0861bbfa0497-kube-api-access-qrvnh\" (UniqueName: \"kubernetes.io/projected/695046b6-8796-44af-bf28-0861bbfa0497-kube-api-access-qrvnh\") pod \"695046b6-8796-44af-bf28-0861bbfa0497\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") "
Nov 03 11:31:50.847332 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.847264    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-openshift-ca\") pod \"695046b6-8796-44af-bf28-0861bbfa0497\" (UID: \"695046b6-8796-44af-bf28-0861bbfa0497\") "
Nov 03 11:31:50.847668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.847643    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-openshift-ca" pod "695046b6-8796-44af-bf28-0861bbfa0497" (UID: "695046b6-8796-44af-bf28-0861bbfa0497"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:50.849240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.849222    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-xks-ca" pod "695046b6-8796-44af-bf28-0861bbfa0497" (UID: "695046b6-8796-44af-bf28-0861bbfa0497"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:50.849484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.849462    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695046b6-8796-44af-bf28-0861bbfa0497-kube-api-access-qrvnh" pod "695046b6-8796-44af-bf28-0861bbfa0497" (UID: "695046b6-8796-44af-bf28-0861bbfa0497"). InnerVolumeSpecName "kube-api-access-qrvnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:50.948151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.948127    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrvnh\" (UniqueName: \"kubernetes.io/projected/695046b6-8796-44af-bf28-0861bbfa0497-kube-api-access-qrvnh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:50.948151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.948152    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:50.948316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:50.948163    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695046b6-8796-44af-bf28-0861bbfa0497-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.081704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.081507    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:51.530873 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.530842    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"
Nov 03 11:31:51.531235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.530844    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q" event={"ID":"695046b6-8796-44af-bf28-0861bbfa0497","Type":"ContainerDied","Data":"01ef956141b1d46356e87a349013ca389017745b0a36a3d1be10a54e417ecedd"}
Nov 03 11:31:51.531235 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.530985    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ef956141b1d46356e87a349013ca389017745b0a36a3d1be10a54e417ecedd"
Nov 03 11:31:51.680763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.680206    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:51.726383 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.726356    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865015    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-kube-api-access-hb8z5\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-kube-api-access-hb8z5\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865074    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-token\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-token\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865120    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-podinfo\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865171    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-envoy\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865333    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759a79a3-3e94-4ac4-8435-b92d3835776c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759a79a3-3e94-4ac4-8435-b92d3835776c-istiod-ca-cert\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865395    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-credential-socket\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865453    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-socket\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865479    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-certs\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.866251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.865515    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-data\") pod \"759a79a3-3e94-4ac4-8435-b92d3835776c\" (UID: \"759a79a3-3e94-4ac4-8435-b92d3835776c\") "
Nov 03 11:31:51.874218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.873796    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-data" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.881598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.881277    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-kube-api-access-hb8z5" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "kube-api-access-hb8z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:51.885111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.884815    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-podinfo" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:31:51.885678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.885474    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-socket" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.886112 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.885856    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-credential-socket" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.886993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.886968    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-certs" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.888995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.888777    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759a79a3-3e94-4ac4-8435-b92d3835776c-istiod-ca-cert" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:51.893026 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.892989    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-envoy" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.901658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.901520    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-token" pod "759a79a3-3e94-4ac4-8435-b92d3835776c" (UID: "759a79a3-3e94-4ac4-8435-b92d3835776c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:51.941008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.940958    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"]
Nov 03 11:31:51.948287 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.948106    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"]
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.967874    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759a79a3-3e94-4ac4-8435-b92d3835776c-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.967907    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.967923    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.967937    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.967951    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.967975    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb8z5\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-kube-api-access-hb8z5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.967993    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.968006    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.968061 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:51.968020    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759a79a3-3e94-4ac4-8435-b92d3835776c-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:52.096646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.096601    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:52.413385 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:52.413293    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:31:52.547296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.547065    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:52.556088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.556056    5817 generic.go:356] "Generic (PLEG): container finished" podID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerID="9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04" exitCode=0
Nov 03 11:31:52.556266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.556173    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"
Nov 03 11:31:52.556266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.556183    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" event={"ID":"759a79a3-3e94-4ac4-8435-b92d3835776c","Type":"ContainerDied","Data":"9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04"}
Nov 03 11:31:52.556266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.556246    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5" event={"ID":"759a79a3-3e94-4ac4-8435-b92d3835776c","Type":"ContainerDied","Data":"e671d52f570dd7ad1181814aab038d746f5ed8ecd0056832d546328d48c50744"}
Nov 03 11:31:52.556405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.556280    5817 scope.go:122] "RemoveContainer" containerID="9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04"
Nov 03 11:31:52.568651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.568634    5817 scope.go:122] "RemoveContainer" containerID="f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc"
Nov 03 11:31:52.576531 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.576507    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"]
Nov 03 11:31:52.580845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.580825    5817 scope.go:122] "RemoveContainer" containerID="9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04"
Nov 03 11:31:52.581228 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:52.581206    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04\": container with ID starting with 9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04 not found: ID does not exist" containerID="9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04"
Nov 03 11:31:52.581342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.581236    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04"} err="failed to get container status \"9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04\": rpc error: code = NotFound desc = could not find container \"9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04\": container with ID starting with 9f1feedb0b004dc501db89189fbfaee0776d40b461ef4c2eaf31f660c24bbd04 not found: ID does not exist"
Nov 03 11:31:52.581342 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.581253    5817 scope.go:122] "RemoveContainer" containerID="f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc"
Nov 03 11:31:52.581560 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:52.581544    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc\": container with ID starting with f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc not found: ID does not exist" containerID="f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc"
Nov 03 11:31:52.581655 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.581565    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc"} err="failed to get container status \"f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc\": rpc error: code = NotFound desc = could not find container \"f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc\": container with ID starting with f2d19fa8ada13bfcf6b3ad6dc8c28388f874cafa94fe76aab2c7de8bfe46e9cc not found: ID does not exist"
Nov 03 11:31:52.582217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:52.582198    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-kpkn5"]
Nov 03 11:31:53.244482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:53.244448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:53.901944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:53.901912    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" path="/var/lib/kubelet/pods/759a79a3-3e94-4ac4-8435-b92d3835776c/volumes"
Nov 03 11:31:53.954284 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:53.954250    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:54.577812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:54.577719    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:54.988306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:54.988269    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:55.141485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.141441    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:55.217934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.217895    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"]
Nov 03 11:31:55.218199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.218170    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy" containerID="cri-o://8241c5a81c01d121196d614163c43c2d884341897c0efad0076b5e82537301dc" gracePeriod=30
Nov 03 11:31:55.303978 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.303875    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:55.404837 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.404805    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"]
Nov 03 11:31:55.405400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405386    5817 cpu_manager.go:414] "containerMap: removing container" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy"
Nov 03 11:31:55.405456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405402    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy"
Nov 03 11:31:55.405456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405417    5817 cpu_manager.go:414] "containerMap: removing container" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy"
Nov 03 11:31:55.405456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405423    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy"
Nov 03 11:31:55.405456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405439    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f043817c-c3d3-4a48-a8eb-de7a02663aad" containerName="revoke-keys"
Nov 03 11:31:55.405456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405444    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f043817c-c3d3-4a48-a8eb-de7a02663aad" containerName="revoke-keys"
Nov 03 11:31:55.405676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405459    5817 cpu_manager.go:414] "containerMap: removing container" podUID="695046b6-8796-44af-bf28-0861bbfa0497" containerName="revoke-keys"
Nov 03 11:31:55.405676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405465    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695046b6-8796-44af-bf28-0861bbfa0497" containerName="revoke-keys"
Nov 03 11:31:55.405676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405542    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy"
Nov 03 11:31:55.405676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405551    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="759a79a3-3e94-4ac4-8435-b92d3835776c" containerName="istio-proxy"
Nov 03 11:31:55.405676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405565    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f043817c-c3d3-4a48-a8eb-de7a02663aad" containerName="revoke-keys"
Nov 03 11:31:55.405676 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.405590    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="695046b6-8796-44af-bf28-0861bbfa0497" containerName="revoke-keys"
Nov 03 11:31:55.410687 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.410669    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.414680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.414645    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:31:55.414838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.414715    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:31:55.416728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.416706    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"]
Nov 03 11:31:55.455200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.455157    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:55.511204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.511174    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84rg\" (UniqueName: \"kubernetes.io/projected/e273cace-a356-4d49-bb5b-4b49b19251bf-kube-api-access-m84rg\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.511361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.511225    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.511524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.511498    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.612585 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.612481    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.612585 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.612553    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84rg\" (UniqueName: \"kubernetes.io/projected/e273cace-a356-4d49-bb5b-4b49b19251bf-kube-api-access-m84rg\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.612818 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.612636    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.613262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.613239    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.615199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.615179    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.620354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.620327    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84rg\" (UniqueName: \"kubernetes.io/projected/e273cace-a356-4d49-bb5b-4b49b19251bf-kube-api-access-m84rg\") pod \"maas-api-revoke-sub-b68cb383-mr45g\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.725524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.725491    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"]
Nov 03 11:31:55.729540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.729515    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:55.729907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.729893    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:55.741951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.741925    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"]
Nov 03 11:31:55.917334 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.917236    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zkz7\" (UniqueName: \"kubernetes.io/projected/a3626712-1259-4178-8fc2-d8222e22f984-kube-api-access-6zkz7\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:55.917334 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.917293    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:55.917547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:55.917498    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:56.018741 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.018709    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:56.019227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.018821    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkz7\" (UniqueName: \"kubernetes.io/projected/a3626712-1259-4178-8fc2-d8222e22f984-kube-api-access-6zkz7\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:56.019227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.018862    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:56.019586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.019547    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:56.021291 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.021267    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:56.031351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.031327    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkz7\" (UniqueName: \"kubernetes.io/projected/a3626712-1259-4178-8fc2-d8222e22f984-kube-api-access-6zkz7\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:56.036606 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.032723    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"]
Nov 03 11:31:56.079035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.079014    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:56.244071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.244033    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:56.358502 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.358475    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"]
Nov 03 11:31:56.358789 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:31:56.358765    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3626712_1259_4178_8fc2_d8222e22f984.slice/crio-e130ce5ac968d0e9508494a42c9de8808a14325e14a4ec8bfa96a8765ae6d0b1 WatchSource:0}: Error finding container e130ce5ac968d0e9508494a42c9de8808a14325e14a4ec8bfa96a8765ae6d0b1: Status 404 returned error can't find the container with id e130ce5ac968d0e9508494a42c9de8808a14325e14a4ec8bfa96a8765ae6d0b1
Nov 03 11:31:56.608862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.608804    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g" event={"ID":"e273cace-a356-4d49-bb5b-4b49b19251bf","Type":"ContainerStarted","Data":"2579ee6c6b6f35a11ee700be1eb62b4a11d321a24d0dc4198fe9770ee5222abe"}
Nov 03 11:31:56.609066 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.608873    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g" event={"ID":"e273cace-a356-4d49-bb5b-4b49b19251bf","Type":"ContainerStarted","Data":"efe7a5d8898bb50021afa16c48885371724234f81016dfcdac3763dd9b9989d4"}
Nov 03 11:31:56.610312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.610286    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g" event={"ID":"a3626712-1259-4178-8fc2-d8222e22f984","Type":"ContainerStarted","Data":"e130ce5ac968d0e9508494a42c9de8808a14325e14a4ec8bfa96a8765ae6d0b1"}
Nov 03 11:31:56.629289 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.629246    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g" podStartSLOduration=1.629230745 podStartE2EDuration="1.629230745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:31:56.626547447 +0000 UTC m=+3746.832129499" watchObservedRunningTime="2026-11-03 11:31:56.629230745 +0000 UTC m=+3746.834812873"
Nov 03 11:31:56.794738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.794707    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh" podUID="dd560ff4-8e15-492e-9278-c636496dd2be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:56.794908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:56.794749    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt" podUID="c6737167-e4c9-4287-bd9e-9151f1f14eed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:57.250484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.250411    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:57.258915 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.258171    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"]
Nov 03 11:31:57.264490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.264464    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"]
Nov 03 11:31:57.395754 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.395709    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:57.621849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.621760    5817 generic.go:356] "Generic (PLEG): container finished" podID="e273cace-a356-4d49-bb5b-4b49b19251bf" containerID="2579ee6c6b6f35a11ee700be1eb62b4a11d321a24d0dc4198fe9770ee5222abe" exitCode=0
Nov 03 11:31:57.621849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.621834    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g" event={"ID":"e273cace-a356-4d49-bb5b-4b49b19251bf","Type":"ContainerDied","Data":"2579ee6c6b6f35a11ee700be1eb62b4a11d321a24d0dc4198fe9770ee5222abe"}
Nov 03 11:31:57.623714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.623687    5817 generic.go:356] "Generic (PLEG): container finished" podID="a3626712-1259-4178-8fc2-d8222e22f984" containerID="2d899571e2c2399eee0facfae978621a6a8a478eff03fa89e124ff4dfdb24b70" exitCode=0
Nov 03 11:31:57.623856 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.623764    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g" event={"ID":"a3626712-1259-4178-8fc2-d8222e22f984","Type":"ContainerDied","Data":"2d899571e2c2399eee0facfae978621a6a8a478eff03fa89e124ff4dfdb24b70"}
Nov 03 11:31:57.971161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:57.971115    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:58.657596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.653726    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:58.824679 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.824650    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:58.855418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.855390    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:58.864451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.864425    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-xks-ca\") pod \"a3626712-1259-4178-8fc2-d8222e22f984\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") "
Nov 03 11:31:58.864630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.864491    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-openshift-ca\") pod \"a3626712-1259-4178-8fc2-d8222e22f984\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") "
Nov 03 11:31:58.864630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.864544    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e273cace-a356-4d49-bb5b-4b49b19251bf-kube-api-access-m84rg\" (UniqueName: \"kubernetes.io/projected/e273cace-a356-4d49-bb5b-4b49b19251bf-kube-api-access-m84rg\") pod \"e273cace-a356-4d49-bb5b-4b49b19251bf\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") "
Nov 03 11:31:58.864630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.864587    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3626712-1259-4178-8fc2-d8222e22f984-kube-api-access-6zkz7\" (UniqueName: \"kubernetes.io/projected/a3626712-1259-4178-8fc2-d8222e22f984-kube-api-access-6zkz7\") pod \"a3626712-1259-4178-8fc2-d8222e22f984\" (UID: \"a3626712-1259-4178-8fc2-d8222e22f984\") "
Nov 03 11:31:58.864843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.864648    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-openshift-ca\") pod \"e273cace-a356-4d49-bb5b-4b49b19251bf\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") "
Nov 03 11:31:58.864843 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.864739    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-xks-ca\") pod \"e273cace-a356-4d49-bb5b-4b49b19251bf\" (UID: \"e273cace-a356-4d49-bb5b-4b49b19251bf\") "
Nov 03 11:31:58.864969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.864930    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-openshift-ca" pod "a3626712-1259-4178-8fc2-d8222e22f984" (UID: "a3626712-1259-4178-8fc2-d8222e22f984"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:58.865290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.865229    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-openshift-ca" pod "e273cace-a356-4d49-bb5b-4b49b19251bf" (UID: "e273cace-a356-4d49-bb5b-4b49b19251bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:58.865290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.865318    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:58.865290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.865340    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:58.867211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.867157    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-xks-ca" pod "e273cace-a356-4d49-bb5b-4b49b19251bf" (UID: "e273cace-a356-4d49-bb5b-4b49b19251bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:58.867704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.867678    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-xks-ca" pod "a3626712-1259-4178-8fc2-d8222e22f984" (UID: "a3626712-1259-4178-8fc2-d8222e22f984"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:58.867817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.867700    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e273cace-a356-4d49-bb5b-4b49b19251bf-kube-api-access-m84rg" pod "e273cace-a356-4d49-bb5b-4b49b19251bf" (UID: "e273cace-a356-4d49-bb5b-4b49b19251bf"). InnerVolumeSpecName "kube-api-access-m84rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:58.868054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.868032    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3626712-1259-4178-8fc2-d8222e22f984-kube-api-access-6zkz7" pod "a3626712-1259-4178-8fc2-d8222e22f984" (UID: "a3626712-1259-4178-8fc2-d8222e22f984"). InnerVolumeSpecName "kube-api-access-6zkz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:58.966625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.966596    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e273cace-a356-4d49-bb5b-4b49b19251bf-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:58.966625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.966622    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3626712-1259-4178-8fc2-d8222e22f984-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:58.966625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.966633    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84rg\" (UniqueName: \"kubernetes.io/projected/e273cace-a356-4d49-bb5b-4b49b19251bf-kube-api-access-m84rg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:58.966891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:58.966648    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zkz7\" (UniqueName: \"kubernetes.io/projected/a3626712-1259-4178-8fc2-d8222e22f984-kube-api-access-6zkz7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:59.295624 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.295556    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 11:31:59.346611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.346538    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:31:59.354680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.354649    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 11:31:59.370640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.370607    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd560ff4-8e15-492e-9278-c636496dd2be-kube-api-access-5p5bs\" (UniqueName: \"kubernetes.io/projected/dd560ff4-8e15-492e-9278-c636496dd2be-kube-api-access-5p5bs\") pod \"dd560ff4-8e15-492e-9278-c636496dd2be\" (UID: \"dd560ff4-8e15-492e-9278-c636496dd2be\") "
Nov 03 11:31:59.370824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.370696    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6737167-e4c9-4287-bd9e-9151f1f14eed-kube-api-access-g7kf2\" (UniqueName: \"kubernetes.io/projected/c6737167-e4c9-4287-bd9e-9151f1f14eed-kube-api-access-g7kf2\") pod \"c6737167-e4c9-4287-bd9e-9151f1f14eed\" (UID: \"c6737167-e4c9-4287-bd9e-9151f1f14eed\") "
Nov 03 11:31:59.370824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.370765    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd560ff4-8e15-492e-9278-c636496dd2be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd560ff4-8e15-492e-9278-c636496dd2be-plugins-config-volume\") pod \"dd560ff4-8e15-492e-9278-c636496dd2be\" (UID: \"dd560ff4-8e15-492e-9278-c636496dd2be\") "
Nov 03 11:31:59.370983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.370834    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6737167-e4c9-4287-bd9e-9151f1f14eed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6737167-e4c9-4287-bd9e-9151f1f14eed-plugins-config-volume\") pod \"c6737167-e4c9-4287-bd9e-9151f1f14eed\" (UID: \"c6737167-e4c9-4287-bd9e-9151f1f14eed\") "
Nov 03 11:31:59.373002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.372959    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd560ff4-8e15-492e-9278-c636496dd2be-plugins-config-volume" pod "dd560ff4-8e15-492e-9278-c636496dd2be" (UID: "dd560ff4-8e15-492e-9278-c636496dd2be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:59.373152 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.373012    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6737167-e4c9-4287-bd9e-9151f1f14eed-plugins-config-volume" pod "c6737167-e4c9-4287-bd9e-9151f1f14eed" (UID: "c6737167-e4c9-4287-bd9e-9151f1f14eed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:59.374723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.373795    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6737167-e4c9-4287-bd9e-9151f1f14eed-kube-api-access-g7kf2" pod "c6737167-e4c9-4287-bd9e-9151f1f14eed" (UID: "c6737167-e4c9-4287-bd9e-9151f1f14eed"). InnerVolumeSpecName "kube-api-access-g7kf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:59.378253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.378227    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd560ff4-8e15-492e-9278-c636496dd2be-kube-api-access-5p5bs" pod "dd560ff4-8e15-492e-9278-c636496dd2be" (UID: "dd560ff4-8e15-492e-9278-c636496dd2be"). InnerVolumeSpecName "kube-api-access-5p5bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:59.473405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.473378    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p5bs\" (UniqueName: \"kubernetes.io/projected/dd560ff4-8e15-492e-9278-c636496dd2be-kube-api-access-5p5bs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:59.473405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.473402    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7kf2\" (UniqueName: \"kubernetes.io/projected/c6737167-e4c9-4287-bd9e-9151f1f14eed-kube-api-access-g7kf2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:59.473405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.473413    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd560ff4-8e15-492e-9278-c636496dd2be-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:59.473657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.473423    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6737167-e4c9-4287-bd9e-9151f1f14eed-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:59.645248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.645154    5817 generic.go:356] "Generic (PLEG): container finished" podID="c6737167-e4c9-4287-bd9e-9151f1f14eed" containerID="34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470" exitCode=0
Nov 03 11:31:59.645248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.645204    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"
Nov 03 11:31:59.645248 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.645234    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt" event={"ID":"c6737167-e4c9-4287-bd9e-9151f1f14eed","Type":"ContainerDied","Data":"34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470"}
Nov 03 11:31:59.645547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.645300    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt" event={"ID":"c6737167-e4c9-4287-bd9e-9151f1f14eed","Type":"ContainerDied","Data":"9044904adc94ffe079d5494346f8497a6401cb6184f505cb42a88b61f11292f1"}
Nov 03 11:31:59.645547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.645334    5817 scope.go:122] "RemoveContainer" containerID="34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470"
Nov 03 11:31:59.646941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.646776    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"
Nov 03 11:31:59.646941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.646815    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g" event={"ID":"a3626712-1259-4178-8fc2-d8222e22f984","Type":"ContainerDied","Data":"e130ce5ac968d0e9508494a42c9de8808a14325e14a4ec8bfa96a8765ae6d0b1"}
Nov 03 11:31:59.646941 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.646859    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e130ce5ac968d0e9508494a42c9de8808a14325e14a4ec8bfa96a8765ae6d0b1"
Nov 03 11:31:59.648490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.648468    5817 generic.go:356] "Generic (PLEG): container finished" podID="dd560ff4-8e15-492e-9278-c636496dd2be" containerID="27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe" exitCode=0
Nov 03 11:31:59.648633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.648524    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"
Nov 03 11:31:59.648633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.648582    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh" event={"ID":"dd560ff4-8e15-492e-9278-c636496dd2be","Type":"ContainerDied","Data":"27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe"}
Nov 03 11:31:59.648633 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.648625    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh" event={"ID":"dd560ff4-8e15-492e-9278-c636496dd2be","Type":"ContainerDied","Data":"3be503fd6003d46460f8ab436dab6c86709dfc01f33d6c440b2271aa613ae8d2"}
Nov 03 11:31:59.650995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.650974    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g" event={"ID":"e273cace-a356-4d49-bb5b-4b49b19251bf","Type":"ContainerDied","Data":"efe7a5d8898bb50021afa16c48885371724234f81016dfcdac3763dd9b9989d4"}
Nov 03 11:31:59.650995 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.650993    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68cb383-mr45g"
Nov 03 11:31:59.651165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.651012    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efe7a5d8898bb50021afa16c48885371724234f81016dfcdac3763dd9b9989d4"
Nov 03 11:31:59.660195 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.660177    5817 scope.go:122] "RemoveContainer" containerID="34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470"
Nov 03 11:31:59.660608 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:59.660440    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470\": container with ID starting with 34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470 not found: ID does not exist" containerID="34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470"
Nov 03 11:31:59.660608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.660465    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470"} err="failed to get container status \"34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470\": rpc error: code = NotFound desc = could not find container \"34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470\": container with ID starting with 34a4ef207927a0d9ec22e7b5c49026655846183114b8b819e72781d928314470 not found: ID does not exist"
Nov 03 11:31:59.660608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.660484    5817 scope.go:122] "RemoveContainer" containerID="27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe"
Nov 03 11:31:59.671181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.671163    5817 scope.go:122] "RemoveContainer" containerID="27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe"
Nov 03 11:31:59.671445 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:31:59.671427    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe\": container with ID starting with 27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe not found: ID does not exist" containerID="27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe"
Nov 03 11:31:59.671497 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.671451    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe"} err="failed to get container status \"27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe\": rpc error: code = NotFound desc = could not find container \"27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe\": container with ID starting with 27313e5bbe6a509c87a5c1fb4a57e9eef8da6183fa260604ef4ffa02ddeafcbe not found: ID does not exist"
Nov 03 11:31:59.678420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.678399    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"]
Nov 03 11:31:59.681816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.681797    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ef1662-7994fffb75-ktcdt"]
Nov 03 11:31:59.699510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.699488    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"]
Nov 03 11:31:59.704692 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.704672    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ef1662-5fc9b869d7-xzvrh"]
Nov 03 11:31:59.904274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.904197    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6737167-e4c9-4287-bd9e-9151f1f14eed" path="/var/lib/kubelet/pods/c6737167-e4c9-4287-bd9e-9151f1f14eed/volumes"
Nov 03 11:31:59.904689 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:31:59.904674    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd560ff4-8e15-492e-9278-c636496dd2be" path="/var/lib/kubelet/pods/dd560ff4-8e15-492e-9278-c636496dd2be/volumes"
Nov 03 11:32:00.010640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.002683    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:00.669303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.669211    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:00.674815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.674791    5817 generic.go:356] "Generic (PLEG): container finished" podID="77c02686-3aa2-4412-8738-4929000230a0" containerID="8241c5a81c01d121196d614163c43c2d884341897c0efad0076b5e82537301dc" exitCode=0
Nov 03 11:32:00.674963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.674841    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" event={"ID":"77c02686-3aa2-4412-8738-4929000230a0","Type":"ContainerDied","Data":"8241c5a81c01d121196d614163c43c2d884341897c0efad0076b5e82537301dc"}
Nov 03 11:32:00.674963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.674899    5817 scope.go:122] "RemoveContainer" containerID="4ca92cc1cbc4d123120205e6fa3855e9fb5dee81d91e3921cf9b779e970da30b"
Nov 03 11:32:00.889135 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.889104    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:32:00.990035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990003    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-credential-socket\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990068    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-socket\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990117    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-kube-api-access-7n487\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-kube-api-access-7n487\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990184    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77c02686-3aa2-4412-8738-4929000230a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c02686-3aa2-4412-8738-4929000230a0-istio-podinfo\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990231    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c02686-3aa2-4412-8738-4929000230a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c02686-3aa2-4412-8738-4929000230a0-istiod-ca-cert\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990262    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-credential-socket" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:00.990505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990327    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-data\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990379    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-istio-token\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-istio-token\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990403    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-socket" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:00.990505 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990410    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-certs\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990513    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-envoy\") pod \"77c02686-3aa2-4412-8738-4929000230a0\" (UID: \"77c02686-3aa2-4412-8738-4929000230a0\") "
Nov 03 11:32:00.990848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990640    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-certs" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:00.990848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990694    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c02686-3aa2-4412-8738-4929000230a0-istiod-ca-cert" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:00.990848 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.990820    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-data" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:00.991393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.991137    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:00.991393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.991158    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:00.991393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.991173    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c02686-3aa2-4412-8738-4929000230a0-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:00.991393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.991189    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:00.991393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.991203    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:00.993226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.993019    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-kube-api-access-7n487" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "kube-api-access-7n487". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:00.993226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.993053    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77c02686-3aa2-4412-8738-4929000230a0-istio-podinfo" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:00.993405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.993375    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-istio-token" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:00.993695 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:00.993673    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-envoy" pod "77c02686-3aa2-4412-8738-4929000230a0" (UID: "77c02686-3aa2-4412-8738-4929000230a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:01.092510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.092480    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:01.092510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.092509    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c02686-3aa2-4412-8738-4929000230a0-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:01.092728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.092520    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n487\" (UniqueName: \"kubernetes.io/projected/77c02686-3aa2-4412-8738-4929000230a0-kube-api-access-7n487\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:01.092728 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.092530    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c02686-3aa2-4412-8738-4929000230a0-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:01.193824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.193790    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:32:01.274904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.270058    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:01.748652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.743408    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn" event={"ID":"77c02686-3aa2-4412-8738-4929000230a0","Type":"ContainerDied","Data":"858429afa70cb066d653e0415d060c047f2c9bb83f02f93119b8cc269c06f077"}
Nov 03 11:32:01.748652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.743475    5817 scope.go:122] "RemoveContainer" containerID="8241c5a81c01d121196d614163c43c2d884341897c0efad0076b5e82537301dc"
Nov 03 11:32:01.748652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.743663    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"
Nov 03 11:32:01.777100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.777037    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"]
Nov 03 11:32:01.785417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.785389    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-d9gzn"]
Nov 03 11:32:01.803644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.803614    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"]
Nov 03 11:32:01.804156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804138    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c6737167-e4c9-4287-bd9e-9151f1f14eed" containerName="payload-pre-processing"
Nov 03 11:32:01.804156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804153    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6737167-e4c9-4287-bd9e-9151f1f14eed" containerName="payload-pre-processing"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804165    5817 cpu_manager.go:414] "containerMap: removing container" podUID="dd560ff4-8e15-492e-9278-c636496dd2be" containerName="payload-processing"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804170    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd560ff4-8e15-492e-9278-c636496dd2be" containerName="payload-processing"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804183    5817 cpu_manager.go:414] "containerMap: removing container" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804188    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804197    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a3626712-1259-4178-8fc2-d8222e22f984" containerName="revoke-keys"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804203    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3626712-1259-4178-8fc2-d8222e22f984" containerName="revoke-keys"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804211    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e273cace-a356-4d49-bb5b-4b49b19251bf" containerName="revoke-keys"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804215    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e273cace-a356-4d49-bb5b-4b49b19251bf" containerName="revoke-keys"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804221    5817 cpu_manager.go:414] "containerMap: removing container" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy"
Nov 03 11:32:01.804301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804226    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy"
Nov 03 11:32:01.804921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804312    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6737167-e4c9-4287-bd9e-9151f1f14eed" containerName="payload-pre-processing"
Nov 03 11:32:01.804921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804319    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd560ff4-8e15-492e-9278-c636496dd2be" containerName="payload-processing"
Nov 03 11:32:01.804921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804327    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy"
Nov 03 11:32:01.804921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804333    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3626712-1259-4178-8fc2-d8222e22f984" containerName="revoke-keys"
Nov 03 11:32:01.804921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804340    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e273cace-a356-4d49-bb5b-4b49b19251bf" containerName="revoke-keys"
Nov 03 11:32:01.804921 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.804344    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c02686-3aa2-4412-8738-4929000230a0" containerName="istio-proxy"
Nov 03 11:32:01.809330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.809290    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:01.818244 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.817858    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:32:01.818638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.818618    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:32:01.822418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.822398    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"]
Nov 03 11:32:01.826427 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.826403    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"]
Nov 03 11:32:01.831547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.831523    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:01.841711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.840391    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"]
Nov 03 11:32:01.902874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.902838    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c02686-3aa2-4412-8738-4929000230a0" path="/var/lib/kubelet/pods/77c02686-3aa2-4412-8738-4929000230a0/volumes"
Nov 03 11:32:01.903480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.903464    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:01.903534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.903495    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:01.903693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.903674    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:01.903743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.903729    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:01.903778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.903755    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pt4h\" (UniqueName: \"kubernetes.io/projected/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-kube-api-access-4pt4h\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:01.903857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:01.903839    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkxw\" (UniqueName: \"kubernetes.io/projected/72a2f80e-8e65-40a7-b522-d6f69edd16f9-kube-api-access-qwkxw\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:02.005627 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.005497    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkxw\" (UniqueName: \"kubernetes.io/projected/72a2f80e-8e65-40a7-b522-d6f69edd16f9-kube-api-access-qwkxw\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:02.006022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.005986    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:02.006199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.006182    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:02.006478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.006460    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:02.006647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.006631    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:02.006817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.006800    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pt4h\" (UniqueName: \"kubernetes.io/projected/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-kube-api-access-4pt4h\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:02.008400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.008375    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:02.008540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.008515    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:02.011021 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.010995    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:02.012222 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.012199    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:02.020199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.020176    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pt4h\" (UniqueName: \"kubernetes.io/projected/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-kube-api-access-4pt4h\") pod \"maas-api-revoke-sub-2115d230-h42wd\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:02.021549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.021531    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkxw\" (UniqueName: \"kubernetes.io/projected/72a2f80e-8e65-40a7-b522-d6f69edd16f9-kube-api-access-qwkxw\") pod \"maas-api-revoke-sub-58f9cb61-xs5p8\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:02.138753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.138704    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:02.139913 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.139883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:02.149610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.149557    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:02.261056 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.260232    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt" podUID="2d28be58-89ea-4734-a9a7-f11bbd106db2" containerName="payload-processing" containerID="cri-o://c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3" gracePeriod=25
Nov 03 11:32:02.265250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.265065    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt" podUID="c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" containerName="payload-pre-processing" containerID="cri-o://b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15" gracePeriod=25
Nov 03 11:32:02.452747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.452712    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"]
Nov 03 11:32:02.453269 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:32:02.453244    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de3163e_4bba_4211_99e7_3c32f6eb8d2d.slice/crio-3e234a3c0907c5d262bcaccc0b373c262b27616f986523e0fd5850c55c5ab55a WatchSource:0}: Error finding container 3e234a3c0907c5d262bcaccc0b373c262b27616f986523e0fd5850c55c5ab55a: Status 404 returned error can't find the container with id 3e234a3c0907c5d262bcaccc0b373c262b27616f986523e0fd5850c55c5ab55a
Nov 03 11:32:02.486744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.486720    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"]
Nov 03 11:32:02.486862 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:32:02.486721    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a2f80e_8e65_40a7_b522_d6f69edd16f9.slice/crio-d3a3dc033c5652d058a4b3216a6580910c9efdc3c35b5ffa0003de8347a9743e WatchSource:0}: Error finding container d3a3dc033c5652d058a4b3216a6580910c9efdc3c35b5ffa0003de8347a9743e: Status 404 returned error can't find the container with id d3a3dc033c5652d058a4b3216a6580910c9efdc3c35b5ffa0003de8347a9743e
Nov 03 11:32:02.768821 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.768777    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd" event={"ID":"5de3163e-4bba-4211-99e7-3c32f6eb8d2d","Type":"ContainerStarted","Data":"3e234a3c0907c5d262bcaccc0b373c262b27616f986523e0fd5850c55c5ab55a"}
Nov 03 11:32:02.770535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.770460    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8" event={"ID":"72a2f80e-8e65-40a7-b522-d6f69edd16f9","Type":"ContainerStarted","Data":"d3a3dc033c5652d058a4b3216a6580910c9efdc3c35b5ffa0003de8347a9743e"}
Nov 03 11:32:02.812968 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:02.812940    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:03.383950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:03.383907    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:03.779816 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:03.779781    5817 generic.go:356] "Generic (PLEG): container finished" podID="5de3163e-4bba-4211-99e7-3c32f6eb8d2d" containerID="9008ab277c73faee72212cf3151194603b39f04c19be48e44b1adcb8326c86d0" exitCode=0
Nov 03 11:32:03.780262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:03.779819    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd" event={"ID":"5de3163e-4bba-4211-99e7-3c32f6eb8d2d","Type":"ContainerDied","Data":"9008ab277c73faee72212cf3151194603b39f04c19be48e44b1adcb8326c86d0"}
Nov 03 11:32:03.781549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:03.781530    5817 generic.go:356] "Generic (PLEG): container finished" podID="72a2f80e-8e65-40a7-b522-d6f69edd16f9" containerID="60d2772d3646782c8579916bc846ac8f3fcbc15192303849b6aca5aa822ca0f6" exitCode=0
Nov 03 11:32:03.781702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:03.781606    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8" event={"ID":"72a2f80e-8e65-40a7-b522-d6f69edd16f9","Type":"ContainerDied","Data":"60d2772d3646782c8579916bc846ac8f3fcbc15192303849b6aca5aa822ca0f6"}
Nov 03 11:32:04.099939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:04.099838    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:04.773154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:04.773012    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:04.954069 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:04.953421    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:04.988683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:04.988660    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:05.040906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.040824    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-xks-ca\") pod \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") "
Nov 03 11:32:05.040906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.040899    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-openshift-ca\") pod \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") "
Nov 03 11:32:05.041659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.040998    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-openshift-ca\") pod \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") "
Nov 03 11:32:05.041904 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.041884    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a2f80e-8e65-40a7-b522-d6f69edd16f9-kube-api-access-qwkxw\" (UniqueName: \"kubernetes.io/projected/72a2f80e-8e65-40a7-b522-d6f69edd16f9-kube-api-access-qwkxw\") pod \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") "
Nov 03 11:32:05.042025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.041630    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-openshift-ca" pod "5de3163e-4bba-4211-99e7-3c32f6eb8d2d" (UID: "5de3163e-4bba-4211-99e7-3c32f6eb8d2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:05.042179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.042104    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-openshift-ca" pod "72a2f80e-8e65-40a7-b522-d6f69edd16f9" (UID: "72a2f80e-8e65-40a7-b522-d6f69edd16f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:05.042262 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.042042    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-xks-ca\") pod \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\" (UID: \"72a2f80e-8e65-40a7-b522-d6f69edd16f9\") "
Nov 03 11:32:05.042846 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.042811    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-kube-api-access-4pt4h\" (UniqueName: \"kubernetes.io/projected/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-kube-api-access-4pt4h\") pod \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\" (UID: \"5de3163e-4bba-4211-99e7-3c32f6eb8d2d\") "
Nov 03 11:32:05.043309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.043289    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:05.043379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.043317    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:05.045229 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.045211    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-xks-ca" pod "5de3163e-4bba-4211-99e7-3c32f6eb8d2d" (UID: "5de3163e-4bba-4211-99e7-3c32f6eb8d2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:05.045962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.045938    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-xks-ca" pod "72a2f80e-8e65-40a7-b522-d6f69edd16f9" (UID: "72a2f80e-8e65-40a7-b522-d6f69edd16f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:05.046301 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.046268    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a2f80e-8e65-40a7-b522-d6f69edd16f9-kube-api-access-qwkxw" pod "72a2f80e-8e65-40a7-b522-d6f69edd16f9" (UID: "72a2f80e-8e65-40a7-b522-d6f69edd16f9"). InnerVolumeSpecName "kube-api-access-qwkxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:05.047059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.047036    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-kube-api-access-4pt4h" pod "5de3163e-4bba-4211-99e7-3c32f6eb8d2d" (UID: "5de3163e-4bba-4211-99e7-3c32f6eb8d2d"). InnerVolumeSpecName "kube-api-access-4pt4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:05.143883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.143851    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pt4h\" (UniqueName: \"kubernetes.io/projected/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-kube-api-access-4pt4h\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:05.143883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.143878    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de3163e-4bba-4211-99e7-3c32f6eb8d2d-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:05.143883 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.143889    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkxw\" (UniqueName: \"kubernetes.io/projected/72a2f80e-8e65-40a7-b522-d6f69edd16f9-kube-api-access-qwkxw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:05.144207 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.143898    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a2f80e-8e65-40a7-b522-d6f69edd16f9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:05.369214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.369082    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:05.801032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.801001    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd"
Nov 03 11:32:05.801032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.801022    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2115d230-h42wd" event={"ID":"5de3163e-4bba-4211-99e7-3c32f6eb8d2d","Type":"ContainerDied","Data":"3e234a3c0907c5d262bcaccc0b373c262b27616f986523e0fd5850c55c5ab55a"}
Nov 03 11:32:05.801307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.801103    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e234a3c0907c5d262bcaccc0b373c262b27616f986523e0fd5850c55c5ab55a"
Nov 03 11:32:05.802683 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.802657    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8" event={"ID":"72a2f80e-8e65-40a7-b522-d6f69edd16f9","Type":"ContainerDied","Data":"d3a3dc033c5652d058a4b3216a6580910c9efdc3c35b5ffa0003de8347a9743e"}
Nov 03 11:32:05.802841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.802706    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3a3dc033c5652d058a4b3216a6580910c9efdc3c35b5ffa0003de8347a9743e"
Nov 03 11:32:05.802841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.802667    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f9cb61-xs5p8"
Nov 03 11:32:05.829637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.829612    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"]
Nov 03 11:32:05.949015 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:05.948965    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:06.088735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:06.088640    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:06.296269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:06.296236    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:06.866817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:06.866763    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:06.968313 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:06.968275    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main" containerID="cri-o://96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094" gracePeriod=45
Nov 03 11:32:06.970517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:06.970467    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" containerID="cri-o://cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e" gracePeriod=45
Nov 03 11:32:06.998108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:06.998079    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": dial tcp 10.133.0.98:8000: connect: connection refused"
Nov 03 11:32:07.262557 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.262531    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 11:32:07.266253 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.266229    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 11:32:07.270068 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270045    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-kube-api-access-v2mls\" (UniqueName: \"kubernetes.io/projected/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-kube-api-access-v2mls\") pod \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") "
Nov 03 11:32:07.270208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270108    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tmp-dir\") pod \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") "
Nov 03 11:32:07.270208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270147    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-dshm\") pod \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") "
Nov 03 11:32:07.270208 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270199    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-dshm\") pod \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") "
Nov 03 11:32:07.270389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270268    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tls-certs\") pod \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") "
Nov 03 11:32:07.270389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270312    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tmp-dir" pod "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" (UID: "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.270389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270318    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-home\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-home\") pod \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") "
Nov 03 11:32:07.270608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270389    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-model-cache\") pod \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") "
Nov 03 11:32:07.270608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270501    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-home\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-home\") pod \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") "
Nov 03 11:32:07.270608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270552    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-model-cache\") pod \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") "
Nov 03 11:32:07.270797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270610    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tls-certs\") pod \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") "
Nov 03 11:32:07.270797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270647    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-kube-api-access-njjv7\" (UniqueName: \"kubernetes.io/projected/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-kube-api-access-njjv7\") pod \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\" (UID: \"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09\") "
Nov 03 11:32:07.270797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270703    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tmp-dir\") pod \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\" (UID: \"9feb7ae5-1475-4ffc-a636-2f8dd611ec28\") "
Nov 03 11:32:07.270974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270791    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-home" pod "9feb7ae5-1475-4ffc-a636-2f8dd611ec28" (UID: "9feb7ae5-1475-4ffc-a636-2f8dd611ec28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.270974 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.270919    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-home" pod "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" (UID: "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.271517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.271179    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-model-cache" pod "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" (UID: "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.271517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.271195    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.271517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.271214    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.271517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.271228    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.271806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.271522    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-model-cache" pod "9feb7ae5-1475-4ffc-a636-2f8dd611ec28" (UID: "9feb7ae5-1475-4ffc-a636-2f8dd611ec28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.271806 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.271594    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tmp-dir" pod "9feb7ae5-1475-4ffc-a636-2f8dd611ec28" (UID: "9feb7ae5-1475-4ffc-a636-2f8dd611ec28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.273461 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.273438    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-kube-api-access-v2mls" pod "9feb7ae5-1475-4ffc-a636-2f8dd611ec28" (UID: "9feb7ae5-1475-4ffc-a636-2f8dd611ec28"). InnerVolumeSpecName "kube-api-access-v2mls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:07.273788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.273765    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-dshm" pod "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" (UID: "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.273898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.273816    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-kube-api-access-njjv7" pod "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" (UID: "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09"). InnerVolumeSpecName "kube-api-access-njjv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:07.273959 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.273891    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tls-certs" pod "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" (UID: "3cec8936-ce5f-407e-a9f1-ff6dbfed0e09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:07.274951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.274920    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-dshm" pod "9feb7ae5-1475-4ffc-a636-2f8dd611ec28" (UID: "9feb7ae5-1475-4ffc-a636-2f8dd611ec28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.275256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.275236    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tls-certs" pod "9feb7ae5-1475-4ffc-a636-2f8dd611ec28" (UID: "9feb7ae5-1475-4ffc-a636-2f8dd611ec28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:07.372564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372528    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.372564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372557    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.372564 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372566    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.372905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372598    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.372905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372607    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.372905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372617    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.372905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372627    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njjv7\" (UniqueName: \"kubernetes.io/projected/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09-kube-api-access-njjv7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.372905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372638    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.372905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.372646    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2mls\" (UniqueName: \"kubernetes.io/projected/9feb7ae5-1475-4ffc-a636-2f8dd611ec28-kube-api-access-v2mls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.661307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.661257    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:07.822136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.822099    5817 generic.go:356] "Generic (PLEG): container finished" podID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerID="cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e" exitCode=0
Nov 03 11:32:07.822311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.822149    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"
Nov 03 11:32:07.822311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.822185    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" event={"ID":"9feb7ae5-1475-4ffc-a636-2f8dd611ec28","Type":"ContainerDied","Data":"cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e"}
Nov 03 11:32:07.822311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.822226    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w" event={"ID":"9feb7ae5-1475-4ffc-a636-2f8dd611ec28","Type":"ContainerDied","Data":"b2a1336bab64a82be10b78f9a0ba7898d950f2a24a44f7e37ccd7facc7f121e3"}
Nov 03 11:32:07.822311 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.822248    5817 scope.go:122] "RemoveContainer" containerID="cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e"
Nov 03 11:32:07.823822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.823801    5817 generic.go:356] "Generic (PLEG): container finished" podID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerID="96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094" exitCode=0
Nov 03 11:32:07.823977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.823846    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" event={"ID":"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09","Type":"ContainerDied","Data":"96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094"}
Nov 03 11:32:07.823977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.823853    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"
Nov 03 11:32:07.823977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.823876    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8" event={"ID":"3cec8936-ce5f-407e-a9f1-ff6dbfed0e09","Type":"ContainerDied","Data":"c1e8769c663f95058823c09895e080f57096c75d7ccc59f8df84a80c5f2370e7"}
Nov 03 11:32:07.835724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.835707    5817 scope.go:122] "RemoveContainer" containerID="cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e"
Nov 03 11:32:07.836035 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:32:07.836012    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e\": container with ID starting with cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e not found: ID does not exist" containerID="cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e"
Nov 03 11:32:07.836103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.836040    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e"} err="failed to get container status \"cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e\": rpc error: code = NotFound desc = could not find container \"cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e\": container with ID starting with cfd88ac55ad3f57fa2be6859317b9c0b5a3582262f599b091a8e072107a7a84e not found: ID does not exist"
Nov 03 11:32:07.836103 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.836055    5817 scope.go:122] "RemoveContainer" containerID="96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094"
Nov 03 11:32:07.846506 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.846488    5817 scope.go:122] "RemoveContainer" containerID="96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094"
Nov 03 11:32:07.846780 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:32:07.846762    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094\": container with ID starting with 96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094 not found: ID does not exist" containerID="96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094"
Nov 03 11:32:07.846845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.846789    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094"} err="failed to get container status \"96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094\": rpc error: code = NotFound desc = could not find container \"96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094\": container with ID starting with 96e8f5309886e233f40261c6eb31fd56bad2f56d5151ae3c65cea996662df094 not found: ID does not exist"
Nov 03 11:32:07.857773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.857752    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"]
Nov 03 11:32:07.866748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.866728    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-w0-f78621-kserve-78f698677c-fnr5w"]
Nov 03 11:32:07.888204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.888179    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"]
Nov 03 11:32:07.902473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.902443    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" path="/var/lib/kubelet/pods/9feb7ae5-1475-4ffc-a636-2f8dd611ec28/volumes"
Nov 03 11:32:07.902853 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:07.902836    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/distinct-2-w0-f78621-kserve-5fb76497d6-lf4v8"]
Nov 03 11:32:08.405036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:08.404915    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:08.550638 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:08.548602    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:08.637803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:08.637764    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"]
Nov 03 11:32:08.640910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:08.640890    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6012b5-ptj6g"]
Nov 03 11:32:08.713943 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:08.713904    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:08.857008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:08.856964    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:09.007309 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.007228    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:09.209654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.209616    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:09.319153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319079    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"]
Nov 03 11:32:09.319620 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319604    5817 cpu_manager.go:414] "containerMap: removing container" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main"
Nov 03 11:32:09.319678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319623    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main"
Nov 03 11:32:09.319678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319638    5817 cpu_manager.go:414] "containerMap: removing container" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main"
Nov 03 11:32:09.319678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319644    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main"
Nov 03 11:32:09.319678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319666    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5de3163e-4bba-4211-99e7-3c32f6eb8d2d" containerName="revoke-keys"
Nov 03 11:32:09.319678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319672    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de3163e-4bba-4211-99e7-3c32f6eb8d2d" containerName="revoke-keys"
Nov 03 11:32:09.319857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319699    5817 cpu_manager.go:414] "containerMap: removing container" podUID="72a2f80e-8e65-40a7-b522-d6f69edd16f9" containerName="revoke-keys"
Nov 03 11:32:09.319857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319704    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a2f80e-8e65-40a7-b522-d6f69edd16f9" containerName="revoke-keys"
Nov 03 11:32:09.319857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319784    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a2f80e-8e65-40a7-b522-d6f69edd16f9" containerName="revoke-keys"
Nov 03 11:32:09.319857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319793    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="9feb7ae5-1475-4ffc-a636-2f8dd611ec28" containerName="main"
Nov 03 11:32:09.319857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319800    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" containerName="main"
Nov 03 11:32:09.319857 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.319809    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de3163e-4bba-4211-99e7-3c32f6eb8d2d" containerName="revoke-keys"
Nov 03 11:32:09.323189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.323171    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.335186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.335161    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:32:09.335491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.335471    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:32:09.335929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.335904    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"]
Nov 03 11:32:09.366725 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.366691    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:09.395647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.395621    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.395758 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.395681    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.395808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.395764    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsntq\" (UniqueName: \"kubernetes.io/projected/70904710-2a21-44a2-b401-c3f930036e7e-kube-api-access-tsntq\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.496698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.496671    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsntq\" (UniqueName: \"kubernetes.io/projected/70904710-2a21-44a2-b401-c3f930036e7e-kube-api-access-tsntq\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.497073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.496776    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.497073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.496812    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.497401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.497384    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.499401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.499380    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.504934 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.504908    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsntq\" (UniqueName: \"kubernetes.io/projected/70904710-2a21-44a2-b401-c3f930036e7e-kube-api-access-tsntq\") pod \"maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.641123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.641049    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:09.902753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.902665    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cec8936-ce5f-407e-a9f1-ff6dbfed0e09" path="/var/lib/kubelet/pods/3cec8936-ce5f-407e-a9f1-ff6dbfed0e09/volumes"
Nov 03 11:32:09.903141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.903124    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3626712-1259-4178-8fc2-d8222e22f984" path="/var/lib/kubelet/pods/a3626712-1259-4178-8fc2-d8222e22f984/volumes"
Nov 03 11:32:09.926036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:09.926006    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"]
Nov 03 11:32:09.926771 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:32:09.926748    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70904710_2a21_44a2_b401_c3f930036e7e.slice/crio-fe1fe0483bc094bf95e2397dc5379f8322fe53c3f7655fa00bb81c2beaaac1f6 WatchSource:0}: Error finding container fe1fe0483bc094bf95e2397dc5379f8322fe53c3f7655fa00bb81c2beaaac1f6: Status 404 returned error can't find the container with id fe1fe0483bc094bf95e2397dc5379f8322fe53c3f7655fa00bb81c2beaaac1f6
Nov 03 11:32:10.020813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:10.020780    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:10.623623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:10.623470    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:10.829899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:10.829865    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp" podUID="bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" containerName="payload-processing" containerID="cri-o://a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513" gracePeriod=25
Nov 03 11:32:10.853014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:10.852981    5817 generic.go:356] "Generic (PLEG): container finished" podID="70904710-2a21-44a2-b401-c3f930036e7e" containerID="a94bf04afa1cce6b97ad9bf11c1f13fe9daa4900cf7a00205c4b239d7ace666a" exitCode=0
Nov 03 11:32:10.853165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:10.853032    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k" event={"ID":"70904710-2a21-44a2-b401-c3f930036e7e","Type":"ContainerDied","Data":"a94bf04afa1cce6b97ad9bf11c1f13fe9daa4900cf7a00205c4b239d7ace666a"}
Nov 03 11:32:10.853165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:10.853103    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k" event={"ID":"70904710-2a21-44a2-b401-c3f930036e7e","Type":"ContainerStarted","Data":"fe1fe0483bc094bf95e2397dc5379f8322fe53c3f7655fa00bb81c2beaaac1f6"}
Nov 03 11:32:11.024447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:11.024408    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt" podUID="2d28be58-89ea-4734-a9a7-f11bbd106db2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:11.335428 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:11.334736    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:12.003524 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.003501    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:12.027420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.027391    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-openshift-ca\") pod \"70904710-2a21-44a2-b401-c3f930036e7e\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") "
Nov 03 11:32:12.027563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.027463    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70904710-2a21-44a2-b401-c3f930036e7e-kube-api-access-tsntq\" (UniqueName: \"kubernetes.io/projected/70904710-2a21-44a2-b401-c3f930036e7e-kube-api-access-tsntq\") pod \"70904710-2a21-44a2-b401-c3f930036e7e\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") "
Nov 03 11:32:12.027723 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.027629    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-xks-ca\") pod \"70904710-2a21-44a2-b401-c3f930036e7e\" (UID: \"70904710-2a21-44a2-b401-c3f930036e7e\") "
Nov 03 11:32:12.028276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.028246    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-openshift-ca" pod "70904710-2a21-44a2-b401-c3f930036e7e" (UID: "70904710-2a21-44a2-b401-c3f930036e7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:12.030492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.030467    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70904710-2a21-44a2-b401-c3f930036e7e-kube-api-access-tsntq" pod "70904710-2a21-44a2-b401-c3f930036e7e" (UID: "70904710-2a21-44a2-b401-c3f930036e7e"). InnerVolumeSpecName "kube-api-access-tsntq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:12.030678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.030528    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-xks-ca" pod "70904710-2a21-44a2-b401-c3f930036e7e" (UID: "70904710-2a21-44a2-b401-c3f930036e7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:12.032201 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.032168    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt" podUID="c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:12.068143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.068112    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:12.128954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.128877    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:12.128954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.128914    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsntq\" (UniqueName: \"kubernetes.io/projected/70904710-2a21-44a2-b401-c3f930036e7e-kube-api-access-tsntq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:12.128954 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.128930    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70904710-2a21-44a2-b401-c3f930036e7e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:12.200034 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.199997    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:12.329077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.329040    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:12.871067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.871037    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"
Nov 03 11:32:12.871249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.871032    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k" event={"ID":"70904710-2a21-44a2-b401-c3f930036e7e","Type":"ContainerDied","Data":"fe1fe0483bc094bf95e2397dc5379f8322fe53c3f7655fa00bb81c2beaaac1f6"}
Nov 03 11:32:12.871249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.871167    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe1fe0483bc094bf95e2397dc5379f8322fe53c3f7655fa00bb81c2beaaac1f6"
Nov 03 11:32:12.973199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:12.973166    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:13.638884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.637768    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"]
Nov 03 11:32:13.638884 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.638026    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" containerID="cri-o://dc0693e46931adce7de223b178ff6df4b13c4f3a51c614867fa6e7fcf0c78646" gracePeriod=30
Nov 03 11:32:13.653899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.653454    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:13.659254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.657956    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"]
Nov 03 11:32:13.659254 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.658228    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" containerID="cri-o://8286f6563e8d1e164681030ae31a725c9653a2a3467a670beec7856f03abd222" gracePeriod=30
Nov 03 11:32:13.677924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.677890    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"]
Nov 03 11:32:13.678128 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.678087    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy" containerID="cri-o://9148b9b95108de1fd3317e6c1bdd88def9cac2bf61e9a472be8bff59347be07f" gracePeriod=30
Nov 03 11:32:13.719677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.719395    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vkhvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:32:13.719677 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:13.719450    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:32:14.350451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:14.350421    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:14.423936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:14.423899    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"]
Nov 03 11:32:14.429179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:14.429150    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f78621-spl5q"]
Nov 03 11:32:14.507594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:14.506633    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:14.663669 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:14.658285    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:14.989631 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:14.989599    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:15.541604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:15.541253    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:15.884166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:15.884089    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp" podUID="bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:15.901805 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:15.901776    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695046b6-8796-44af-bf28-0861bbfa0497" path="/var/lib/kubelet/pods/695046b6-8796-44af-bf28-0861bbfa0497/volumes"
Nov 03 11:32:16.369647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:16.369238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:17.060657 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:17.060614    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:17.312595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:17.312468    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:17.471164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:17.471131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:17.661371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:17.661278    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:17.814924 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:17.814325    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:18.001404 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.001370    5817 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b9tw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:32:18.001601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.001446    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:32:18.056374 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.056341    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"]
Nov 03 11:32:18.056902 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.056888    5817 cpu_manager.go:414] "containerMap: removing container" podUID="70904710-2a21-44a2-b401-c3f930036e7e" containerName="revoke-keys"
Nov 03 11:32:18.056962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.056903    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70904710-2a21-44a2-b401-c3f930036e7e" containerName="revoke-keys"
Nov 03 11:32:18.057010 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.057005    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="70904710-2a21-44a2-b401-c3f930036e7e" containerName="revoke-keys"
Nov 03 11:32:18.060748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.060730    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.071370 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.071341    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"]
Nov 03 11:32:18.082438 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.082415    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:32:18.082544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.082493    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:32:18.129055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.129028    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsqks\" (UniqueName: \"kubernetes.io/projected/a101e7f9-0f65-471a-bdcb-1d70fad062d9-kube-api-access-bsqks\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.129257 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.129234    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.129316 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.129306    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.230084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.230054    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.230249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.230094    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.230249 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.230122    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsqks\" (UniqueName: \"kubernetes.io/projected/a101e7f9-0f65-471a-bdcb-1d70fad062d9-kube-api-access-bsqks\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.230842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.230818    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.232810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.232785    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.237951 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.237928    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsqks\" (UniqueName: \"kubernetes.io/projected/a101e7f9-0f65-471a-bdcb-1d70fad062d9-kube-api-access-bsqks\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.378748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.378671    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:18.628947 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.628844    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:18.663609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.663567    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"]
Nov 03 11:32:18.665474 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:32:18.665446    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda101e7f9_0f65_471a_bdcb_1d70fad062d9.slice/crio-3d21a19235f911bad37711a565b8c89b07a41308e30ea3e6c431503f4bd1afe6 WatchSource:0}: Error finding container 3d21a19235f911bad37711a565b8c89b07a41308e30ea3e6c431503f4bd1afe6: Status 404 returned error can't find the container with id 3d21a19235f911bad37711a565b8c89b07a41308e30ea3e6c431503f4bd1afe6
Nov 03 11:32:18.923737 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.923704    5817 generic.go:356] "Generic (PLEG): container finished" podID="e6723334-88fc-4f50-93be-7c4af48df73f" containerID="8286f6563e8d1e164681030ae31a725c9653a2a3467a670beec7856f03abd222" exitCode=0
Nov 03 11:32:18.924035 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.924016    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" event={"ID":"e6723334-88fc-4f50-93be-7c4af48df73f","Type":"ContainerDied","Data":"8286f6563e8d1e164681030ae31a725c9653a2a3467a670beec7856f03abd222"}
Nov 03 11:32:18.927897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.927870    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs" event={"ID":"a101e7f9-0f65-471a-bdcb-1d70fad062d9","Type":"ContainerStarted","Data":"3d21a19235f911bad37711a565b8c89b07a41308e30ea3e6c431503f4bd1afe6"}
Nov 03 11:32:18.933989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.933832    5817 generic.go:356] "Generic (PLEG): container finished" podID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerID="9148b9b95108de1fd3317e6c1bdd88def9cac2bf61e9a472be8bff59347be07f" exitCode=0
Nov 03 11:32:18.933989 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.933920    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" event={"ID":"0908808c-9d5c-4748-a226-c73289d2b7ed","Type":"ContainerDied","Data":"9148b9b95108de1fd3317e6c1bdd88def9cac2bf61e9a472be8bff59347be07f"}
Nov 03 11:32:18.936436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.936301    5817 generic.go:356] "Generic (PLEG): container finished" podID="51fec13f-2b85-413d-88a6-de064830453b" containerID="dc0693e46931adce7de223b178ff6df4b13c4f3a51c614867fa6e7fcf0c78646" exitCode=0
Nov 03 11:32:18.936436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.936391    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" event={"ID":"51fec13f-2b85-413d-88a6-de064830453b","Type":"ContainerDied","Data":"dc0693e46931adce7de223b178ff6df4b13c4f3a51c614867fa6e7fcf0c78646"}
Nov 03 11:32:18.948814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:18.947700    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 11:32:19.012379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.012352    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:32:19.016946 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.016924    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:32:19.038558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.038425    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-socket" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.038558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.038497    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-socket\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.039730 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.039709    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-envoy\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.039804 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.039744    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-credential-socket\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.039854 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.039812    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-certs\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.039895 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.039880    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-data\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.040017 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.040000    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51fec13f-2b85-413d-88a6-de064830453b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fec13f-2b85-413d-88a6-de064830453b-istio-podinfo\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.040277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.040258    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-certs" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.040443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.040427    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-credential-socket" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.040651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.040633    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-data" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.041646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.040811    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-kube-api-access-mdcbx\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-kube-api-access-mdcbx\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.041646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.040870    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fec13f-2b85-413d-88a6-de064830453b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fec13f-2b85-413d-88a6-de064830453b-istiod-ca-cert\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.041646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.040944    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-istio-token\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-istio-token\") pod \"51fec13f-2b85-413d-88a6-de064830453b\" (UID: \"51fec13f-2b85-413d-88a6-de064830453b\") "
Nov 03 11:32:19.046594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.042516    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.046594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.042541    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.046594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.042557    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.046594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.042592    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.046594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.043002    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-envoy" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.046594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.043756    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51fec13f-2b85-413d-88a6-de064830453b-istio-podinfo" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:19.046594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.044171    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fec13f-2b85-413d-88a6-de064830453b-istiod-ca-cert" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:19.047036 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.046861    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-kube-api-access-mdcbx" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "kube-api-access-mdcbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:19.050859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.050826    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-istio-token" pod "51fec13f-2b85-413d-88a6-de064830453b" (UID: "51fec13f-2b85-413d-88a6-de064830453b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:19.143961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.143935    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-socket\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.143981    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-certs\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144011    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-istio-token\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-istio-token\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144070    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-envoy\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144099    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-data\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144163    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-credential-socket\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144177    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-socket" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144199    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-credential-socket\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144240    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-certs\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144244    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-certs" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144286    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-kube-api-access-vqzw4\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-kube-api-access-vqzw4\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144326    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-data\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144359    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-token\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-token\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144390    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6723334-88fc-4f50-93be-7c4af48df73f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723334-88fc-4f50-93be-7c4af48df73f-istiod-ca-cert\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.144387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144380    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-credential-socket" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144425    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-envoy\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144426    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-credential-socket" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144485    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0908808c-9d5c-4748-a226-c73289d2b7ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908808c-9d5c-4748-a226-c73289d2b7ed-istiod-ca-cert\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144520    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-socket\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144550    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-kube-api-access-gtzsr\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-kube-api-access-gtzsr\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144609    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-data" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144610    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-data" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144625    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-certs" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144642    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-podinfo\") pod \"0908808c-9d5c-4748-a226-c73289d2b7ed\" (UID: \"0908808c-9d5c-4748-a226-c73289d2b7ed\") "
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144705    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6723334-88fc-4f50-93be-7c4af48df73f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723334-88fc-4f50-93be-7c4af48df73f-istio-podinfo\") pod \"e6723334-88fc-4f50-93be-7c4af48df73f\" (UID: \"e6723334-88fc-4f50-93be-7c4af48df73f\") "
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144847    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-socket" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.145351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.144955    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0908808c-9d5c-4748-a226-c73289d2b7ed-istiod-ca-cert" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145380    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdcbx\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-kube-api-access-mdcbx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145404    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145423    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fec13f-2b85-413d-88a6-de064830453b-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145439    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145454    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145469    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fec13f-2b85-413d-88a6-de064830453b-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145484    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145498    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fec13f-2b85-413d-88a6-de064830453b-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145498    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6723334-88fc-4f50-93be-7c4af48df73f-istiod-ca-cert" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145512    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145526    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908808c-9d5c-4748-a226-c73289d2b7ed-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145540    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145555    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145587    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.146102 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.145602    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fec13f-2b85-413d-88a6-de064830453b-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.147288 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.147266    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-envoy" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.147418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.147394    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-istio-token" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:19.147744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.147715    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6723334-88fc-4f50-93be-7c4af48df73f-istio-podinfo" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:19.147744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.147737    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-token" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:19.147880 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.147778    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-podinfo" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:19.148117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.148101    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-kube-api-access-vqzw4" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "kube-api-access-vqzw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:19.148165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.148148    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-kube-api-access-gtzsr" pod "0908808c-9d5c-4748-a226-c73289d2b7ed" (UID: "0908808c-9d5c-4748-a226-c73289d2b7ed"). InnerVolumeSpecName "kube-api-access-gtzsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:19.148851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.148837    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-envoy" pod "e6723334-88fc-4f50-93be-7c4af48df73f" (UID: "e6723334-88fc-4f50-93be-7c4af48df73f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:19.246634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246608    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzsr\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-kube-api-access-gtzsr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.246634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246633    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.246814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246643    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723334-88fc-4f50-93be-7c4af48df73f-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.246814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246652    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.246814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246663    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.246814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246678    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqzw4\" (UniqueName: \"kubernetes.io/projected/e6723334-88fc-4f50-93be-7c4af48df73f-kube-api-access-vqzw4\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.246814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246694    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0908808c-9d5c-4748-a226-c73289d2b7ed-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.246814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246707    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723334-88fc-4f50-93be-7c4af48df73f-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.246814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.246722    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723334-88fc-4f50-93be-7c4af48df73f-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:19.458822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.458748    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:19.593443 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.593241    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:19.945690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.945660    5817 generic.go:356] "Generic (PLEG): container finished" podID="a101e7f9-0f65-471a-bdcb-1d70fad062d9" containerID="502b7627c9151b27a3ebc3a0519b6550ca16b0b59efd47475fedd6fa51cb748f" exitCode=0
Nov 03 11:32:19.945819 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.945741    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs" event={"ID":"a101e7f9-0f65-471a-bdcb-1d70fad062d9","Type":"ContainerDied","Data":"502b7627c9151b27a3ebc3a0519b6550ca16b0b59efd47475fedd6fa51cb748f"}
Nov 03 11:32:19.947226 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.947212    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"
Nov 03 11:32:19.947337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.947223    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn" event={"ID":"0908808c-9d5c-4748-a226-c73289d2b7ed","Type":"ContainerDied","Data":"fa45226b1b6e4772163693451925ad9de0f38a0b08eeef845ec4ea86a03c81a6"}
Nov 03 11:32:19.947337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.947270    5817 scope.go:122] "RemoveContainer" containerID="9148b9b95108de1fd3317e6c1bdd88def9cac2bf61e9a472be8bff59347be07f"
Nov 03 11:32:19.949246 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.949225    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"
Nov 03 11:32:19.949352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.949256    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2" event={"ID":"51fec13f-2b85-413d-88a6-de064830453b","Type":"ContainerDied","Data":"a232c93d9953e57931b0d533755fac2cb76d3d7f0aac1dea60f58277affbd015"}
Nov 03 11:32:19.951281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.951262    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"
Nov 03 11:32:19.951464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.951313    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr" event={"ID":"e6723334-88fc-4f50-93be-7c4af48df73f","Type":"ContainerDied","Data":"7fef6aeb3584d87bec21b78258af85dab85885a3b83595df363c6c95883f453c"}
Nov 03 11:32:19.960202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.960165    5817 scope.go:122] "RemoveContainer" containerID="dc0693e46931adce7de223b178ff6df4b13c4f3a51c614867fa6e7fcf0c78646"
Nov 03 11:32:19.970697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.970665    5817 scope.go:122] "RemoveContainer" containerID="8286f6563e8d1e164681030ae31a725c9653a2a3467a670beec7856f03abd222"
Nov 03 11:32:19.978892 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.978864    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"]
Nov 03 11:32:19.982522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.982505    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b9tw2"]
Nov 03 11:32:19.993769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.993748    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"]
Nov 03 11:32:19.998240 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.998220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:19.998768 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:19.998750    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vkhvr"]
Nov 03 11:32:20.011299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:20.011274    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"]
Nov 03 11:32:20.015144 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:20.015123    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7hrjn"]
Nov 03 11:32:20.140487 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:20.140455    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:20.338643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:20.338607    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"]
Nov 03 11:32:20.351085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:20.351052    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-35936b-g2x4k"]
Nov 03 11:32:20.385022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:20.384988    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:20.562710 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:20.562657    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:20.846736 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:20.846704    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:21.023957 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.023916    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt" podUID="2d28be58-89ea-4734-a9a7-f11bbd106db2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:21.134016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.133980    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:21.159877 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.159850    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"]
Nov 03 11:32:21.160426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160408    5817 cpu_manager.go:414] "containerMap: removing container" podUID="a101e7f9-0f65-471a-bdcb-1d70fad062d9" containerName="revoke-keys"
Nov 03 11:32:21.160537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160428    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a101e7f9-0f65-471a-bdcb-1d70fad062d9" containerName="revoke-keys"
Nov 03 11:32:21.160537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160443    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy"
Nov 03 11:32:21.160537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160450    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy"
Nov 03 11:32:21.160537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160465    5817 cpu_manager.go:414] "containerMap: removing container" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy"
Nov 03 11:32:21.160537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160472    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy"
Nov 03 11:32:21.160537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160520    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy"
Nov 03 11:32:21.160537 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160527    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy"
Nov 03 11:32:21.160982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160674    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fec13f-2b85-413d-88a6-de064830453b" containerName="istio-proxy"
Nov 03 11:32:21.160982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160696    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" containerName="istio-proxy"
Nov 03 11:32:21.160982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160711    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="a101e7f9-0f65-471a-bdcb-1d70fad062d9" containerName="revoke-keys"
Nov 03 11:32:21.160982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.160720    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" containerName="istio-proxy"
Nov 03 11:32:21.164234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.164212    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.174704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.174680    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ce5555-openshift-default-dockercfg-2h6ft\""
Nov 03 11:32:21.176054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.176029    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"]
Nov 03 11:32:21.235100 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.235066    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:21.269128 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269098    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-openshift-ca\") pod \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") "
Nov 03 11:32:21.269305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269222    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-xks-ca\") pod \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") "
Nov 03 11:32:21.269305 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269264    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a101e7f9-0f65-471a-bdcb-1d70fad062d9-kube-api-access-bsqks\" (UniqueName: \"kubernetes.io/projected/a101e7f9-0f65-471a-bdcb-1d70fad062d9-kube-api-access-bsqks\") pod \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\" (UID: \"a101e7f9-0f65-471a-bdcb-1d70fad062d9\") "
Nov 03 11:32:21.269484 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269462    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52aa617-4064-486f-a01d-993eb9ef36e5-istiod-ca-cert\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.269563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269496    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-openshift-ca" pod "a101e7f9-0f65-471a-bdcb-1d70fad062d9" (UID: "a101e7f9-0f65-471a-bdcb-1d70fad062d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:21.269680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269658    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.269749 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269731    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-data\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.269801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269786    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-credential-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.269863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269804    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-envoy\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.269942 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269922    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-podinfo\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.270001 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.269983    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89p5c\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-kube-api-access-89p5c\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.270105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.270081    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-certs\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.270166 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.270133    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-token\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.270283 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.270268    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:21.272059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.272031    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-xks-ca" pod "a101e7f9-0f65-471a-bdcb-1d70fad062d9" (UID: "a101e7f9-0f65-471a-bdcb-1d70fad062d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:21.272059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.272054    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a101e7f9-0f65-471a-bdcb-1d70fad062d9-kube-api-access-bsqks" pod "a101e7f9-0f65-471a-bdcb-1d70fad062d9" (UID: "a101e7f9-0f65-471a-bdcb-1d70fad062d9"). InnerVolumeSpecName "kube-api-access-bsqks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:21.371609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.371519    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89p5c\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-kube-api-access-89p5c\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.371609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.371592    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-certs\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.371632    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-token\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.371686    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52aa617-4064-486f-a01d-993eb9ef36e5-istiod-ca-cert\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.371875    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.371949    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-data\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.371988    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-credential-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372015    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-envoy\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372088 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372012    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-certs\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372099    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-podinfo\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372201    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101e7f9-0f65-471a-bdcb-1d70fad062d9-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:21.372468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372219    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsqks\" (UniqueName: \"kubernetes.io/projected/a101e7f9-0f65-471a-bdcb-1d70fad062d9-kube-api-access-bsqks\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:21.372468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372219    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372395    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-credential-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372437    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-data\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.372468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.372449    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52aa617-4064-486f-a01d-993eb9ef36e5-istiod-ca-cert\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.374369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.374345    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-envoy\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.374743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.374727    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-podinfo\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.380784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.380639    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-token\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.382338 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.382305    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89p5c\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-kube-api-access-89p5c\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.407820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.407792    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:21.484462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.484425    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:21.569586 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.569541    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:21.666937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.666861    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"]
Nov 03 11:32:21.667098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.667078    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy" containerID="cri-o://70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116" gracePeriod=30
Nov 03 11:32:21.739554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.739514    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:21.772702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.772677    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"]
Nov 03 11:32:21.773445 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:32:21.773413    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52aa617_4064_486f_a01d_993eb9ef36e5.slice/crio-5b3aff399a6689c8a4516cd13de0f1ed35eb0e7fcbbd3ebcd2458e19722af020 WatchSource:0}: Error finding container 5b3aff399a6689c8a4516cd13de0f1ed35eb0e7fcbbd3ebcd2458e19722af020: Status 404 returned error can't find the container with id 5b3aff399a6689c8a4516cd13de0f1ed35eb0e7fcbbd3ebcd2458e19722af020
Nov 03 11:32:21.776154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.776123    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:32:21.776245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.776187    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:32:21.776245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.776218    5817 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:32:21.907594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.905178    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0908808c-9d5c-4748-a226-c73289d2b7ed" path="/var/lib/kubelet/pods/0908808c-9d5c-4748-a226-c73289d2b7ed/volumes"
Nov 03 11:32:21.907594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.905786    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fec13f-2b85-413d-88a6-de064830453b" path="/var/lib/kubelet/pods/51fec13f-2b85-413d-88a6-de064830453b/volumes"
Nov 03 11:32:21.907594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.906281    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70904710-2a21-44a2-b401-c3f930036e7e" path="/var/lib/kubelet/pods/70904710-2a21-44a2-b401-c3f930036e7e/volumes"
Nov 03 11:32:21.907594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.907244    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6723334-88fc-4f50-93be-7c4af48df73f" path="/var/lib/kubelet/pods/e6723334-88fc-4f50-93be-7c4af48df73f/volumes"
Nov 03 11:32:21.982459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.982418    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" event={"ID":"b52aa617-4064-486f-a01d-993eb9ef36e5","Type":"ContainerStarted","Data":"fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276"}
Nov 03 11:32:21.982711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.982484    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" event={"ID":"b52aa617-4064-486f-a01d-993eb9ef36e5","Type":"ContainerStarted","Data":"5b3aff399a6689c8a4516cd13de0f1ed35eb0e7fcbbd3ebcd2458e19722af020"}
Nov 03 11:32:21.984003 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.983979    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs" event={"ID":"a101e7f9-0f65-471a-bdcb-1d70fad062d9","Type":"ContainerDied","Data":"3d21a19235f911bad37711a565b8c89b07a41308e30ea3e6c431503f4bd1afe6"}
Nov 03 11:32:21.984124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.984011    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"
Nov 03 11:32:21.984124 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:21.984018    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d21a19235f911bad37711a565b8c89b07a41308e30ea3e6c431503f4bd1afe6"
Nov 03 11:32:22.004390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.004332    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podStartSLOduration=1.004314349 podStartE2EDuration="1.004314349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:32:22.0022205 +0000 UTC m=+3772.207802552" watchObservedRunningTime="2026-11-03 11:32:22.004314349 +0000 UTC m=+3772.209896401"
Nov 03 11:32:22.031744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.031712    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt" podUID="c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:22.063756 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.063722    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:22.461455 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:32:22.461418    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:32:22.462219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.461748    5817 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:32:22.462219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.461816    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:32:22.496622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.493429    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:22.496622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.495412    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Nov 03 11:32:22.496622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.495460    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Nov 03 11:32:22.634508 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.634482    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 11:32:22.671492 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.671461    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 11:32:22.722593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.721861    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d28be58-89ea-4734-a9a7-f11bbd106db2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d28be58-89ea-4734-a9a7-f11bbd106db2-plugins-config-volume\") pod \"2d28be58-89ea-4734-a9a7-f11bbd106db2\" (UID: \"2d28be58-89ea-4734-a9a7-f11bbd106db2\") "
Nov 03 11:32:22.722593 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.721928    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d28be58-89ea-4734-a9a7-f11bbd106db2-kube-api-access-xbbvv\" (UniqueName: \"kubernetes.io/projected/2d28be58-89ea-4734-a9a7-f11bbd106db2-kube-api-access-xbbvv\") pod \"2d28be58-89ea-4734-a9a7-f11bbd106db2\" (UID: \"2d28be58-89ea-4734-a9a7-f11bbd106db2\") "
Nov 03 11:32:22.723645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.723200    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d28be58-89ea-4734-a9a7-f11bbd106db2-plugins-config-volume" pod "2d28be58-89ea-4734-a9a7-f11bbd106db2" (UID: "2d28be58-89ea-4734-a9a7-f11bbd106db2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:22.731070 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.731043    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d28be58-89ea-4734-a9a7-f11bbd106db2-kube-api-access-xbbvv" pod "2d28be58-89ea-4734-a9a7-f11bbd106db2" (UID: "2d28be58-89ea-4734-a9a7-f11bbd106db2"). InnerVolumeSpecName "kube-api-access-xbbvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:22.823817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.823723    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-plugins-config-volume\") pod \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\" (UID: \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\") "
Nov 03 11:32:22.823817 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.823790    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-kube-api-access-vkbrr\" (UniqueName: \"kubernetes.io/projected/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-kube-api-access-vkbrr\") pod \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\" (UID: \"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472\") "
Nov 03 11:32:22.824073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.824057    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d28be58-89ea-4734-a9a7-f11bbd106db2-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:22.824073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.824071    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbbvv\" (UniqueName: \"kubernetes.io/projected/2d28be58-89ea-4734-a9a7-f11bbd106db2-kube-api-access-xbbvv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:22.824210 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.824165    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-plugins-config-volume" pod "c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" (UID: "c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:22.826160 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.826137    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-kube-api-access-vkbrr" pod "c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" (UID: "c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472"). InnerVolumeSpecName "kube-api-access-vkbrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:22.871704 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.871670    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:22.925519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.925467    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:22.925519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:22.925509    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbrr\" (UniqueName: \"kubernetes.io/projected/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472-kube-api-access-vkbrr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:23.024902 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.024866    5817 generic.go:356] "Generic (PLEG): container finished" podID="c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" containerID="b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15" exitCode=0
Nov 03 11:32:23.025117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.024919    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"
Nov 03 11:32:23.025117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.025025    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt" event={"ID":"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472","Type":"ContainerDied","Data":"b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15"}
Nov 03 11:32:23.025117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.025073    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt" event={"ID":"c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472","Type":"ContainerDied","Data":"9e8724b26ef9041b2679c59994d4cd03d27ad74224cd9f44530f39e0daa2d151"}
Nov 03 11:32:23.025117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.025110    5817 scope.go:122] "RemoveContainer" containerID="b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15"
Nov 03 11:32:23.027192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.027033    5817 generic.go:356] "Generic (PLEG): container finished" podID="2d28be58-89ea-4734-a9a7-f11bbd106db2" containerID="c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3" exitCode=0
Nov 03 11:32:23.027675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.027631    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt" event={"ID":"2d28be58-89ea-4734-a9a7-f11bbd106db2","Type":"ContainerDied","Data":"c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3"}
Nov 03 11:32:23.027675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.027645    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"
Nov 03 11:32:23.027885 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.027683    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt" event={"ID":"2d28be58-89ea-4734-a9a7-f11bbd106db2","Type":"ContainerDied","Data":"5a5e2d7f960210ce15ee8018b8a12088e71578f683e71f44b00e2ab91cc1d294"}
Nov 03 11:32:23.046206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.046116    5817 scope.go:122] "RemoveContainer" containerID="b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15"
Nov 03 11:32:23.048609 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:32:23.046460    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15\": container with ID starting with b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15 not found: ID does not exist" containerID="b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15"
Nov 03 11:32:23.048609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.046504    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15"} err="failed to get container status \"b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15\": rpc error: code = NotFound desc = could not find container \"b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15\": container with ID starting with b6840ffa59e3a7f056d8b8efa9844169a8f73d7b2aa020158e147503869dfa15 not found: ID does not exist"
Nov 03 11:32:23.048609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.046528    5817 scope.go:122] "RemoveContainer" containerID="c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3"
Nov 03 11:32:23.051849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.051823    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"]
Nov 03 11:32:23.055931 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.055911    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f78621-5c4f6c9d4c-92tkt"]
Nov 03 11:32:23.065999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.065980    5817 scope.go:122] "RemoveContainer" containerID="c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3"
Nov 03 11:32:23.066604 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:32:23.066546    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3\": container with ID starting with c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3 not found: ID does not exist" containerID="c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3"
Nov 03 11:32:23.066718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.066598    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3"} err="failed to get container status \"c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3\": rpc error: code = NotFound desc = could not find container \"c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3\": container with ID starting with c4a22789b54d1cf32bf6264049c56407c46d9cbeeacf34b63524b1591dfc54c3 not found: ID does not exist"
Nov 03 11:32:23.070647 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.070612    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"]
Nov 03 11:32:23.075204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.075153    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f78621-669d499979-mgwqt"]
Nov 03 11:32:23.485562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.485528    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Nov 03 11:32:23.485977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.485634    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Nov 03 11:32:23.635232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.635201    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:23.920120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.920038    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d28be58-89ea-4734-a9a7-f11bbd106db2" path="/var/lib/kubelet/pods/2d28be58-89ea-4734-a9a7-f11bbd106db2/volumes"
Nov 03 11:32:23.920623 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:23.920606    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" path="/var/lib/kubelet/pods/c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472/volumes"
Nov 03 11:32:24.179352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:24.179314    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:24.316598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:24.315999    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:24.484850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:24.484769    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Nov 03 11:32:24.484850 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:24.484823    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Nov 03 11:32:24.935150 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:24.935104    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:25.485698 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:25.485669    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Nov 03 11:32:25.485919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:25.485717    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Nov 03 11:32:25.643237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:25.643190    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:25.884141 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:25.884059    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp" podUID="bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:26.340252 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:26.340160    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:26.485377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:26.485341    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Nov 03 11:32:26.485558 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:26.485394    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Nov 03 11:32:27.015190 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.014813    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 11:32:27.054900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.054795    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:27.066087 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.066060    5817 generic.go:356] "Generic (PLEG): container finished" podID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerID="70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116" exitCode=0
Nov 03 11:32:27.066328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.066110    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"
Nov 03 11:32:27.066383 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.066129    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" event={"ID":"b8a65377-d9b1-4c74-9aa1-ec3c80aea824","Type":"ContainerDied","Data":"70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116"}
Nov 03 11:32:27.066625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.066417    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd" event={"ID":"b8a65377-d9b1-4c74-9aa1-ec3c80aea824","Type":"ContainerDied","Data":"22d1d1164433c93c244f7fb4e5feca8b6544f747e71a2716bcf635dbdd116b24"}
Nov 03 11:32:27.066625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.066460    5817 scope.go:122] "RemoveContainer" containerID="70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116"
Nov 03 11:32:27.078743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.078708    5817 scope.go:122] "RemoveContainer" containerID="70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116"
Nov 03 11:32:27.079104 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:32:27.079063    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116\": container with ID starting with 70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116 not found: ID does not exist" containerID="70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116"
Nov 03 11:32:27.079238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.079102    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116"} err="failed to get container status \"70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116\": rpc error: code = NotFound desc = could not find container \"70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116\": container with ID starting with 70b5dacc3ed4b240cab95ee5eb1d1e0b61468e745eccd6a9717d7cf39b4c0116 not found: ID does not exist"
Nov 03 11:32:27.170644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.170615    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-token\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-token\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.170823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.170659    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-socket\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.170823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.170710    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istiod-ca-cert\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.170823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.170784    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-credential-socket\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.170997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.170875    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-certs\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.170997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.170925    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-envoy\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.170997 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.170957    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-kube-api-access-xv6vp\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-kube-api-access-xv6vp\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.171179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.170996    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-data\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.171179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171026    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-podinfo\") pod \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\" (UID: \"b8a65377-d9b1-4c74-9aa1-ec3c80aea824\") "
Nov 03 11:32:27.171179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171027    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-socket" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.171179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171159    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-credential-socket" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.171430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171198    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istiod-ca-cert" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:27.171430 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171296    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-data" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.173530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171637    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.173530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171666    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.173530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171698    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.173530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171712    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.173530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.171918    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-certs" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.175562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.174384    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-token" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:27.175562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.174474    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-envoy" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.175562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.175530    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-podinfo" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:27.178742 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.178716    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-kube-api-access-xv6vp" pod "b8a65377-d9b1-4c74-9aa1-ec3c80aea824" (UID: "b8a65377-d9b1-4c74-9aa1-ec3c80aea824"). InnerVolumeSpecName "kube-api-access-xv6vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:27.272626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.272510    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.272626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.272541    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.272626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.272556    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6vp\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-kube-api-access-xv6vp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.272626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.272600    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.272626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.272613    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a65377-d9b1-4c74-9aa1-ec3c80aea824-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.417678 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.417630    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"]
Nov 03 11:32:27.423512 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.423481    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-hp2qd"]
Nov 03 11:32:27.485542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.485514    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Nov 03 11:32:27.485700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.485599    5817 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Nov 03 11:32:27.785043 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.784996    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:27.903441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:27.903397    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" path="/var/lib/kubelet/pods/b8a65377-d9b1-4c74-9aa1-ec3c80aea824/volumes"
Nov 03 11:32:28.174456 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.174375    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"]
Nov 03 11:32:28.243468 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.243430    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:28.395809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.395766    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:28.490067 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.490040    5817 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:28.490471 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.490451    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:28.491525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.491503    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:32:28.518482 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.518454    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:32:28.582600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.581806    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:28.747609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:28.747497    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:29.147684 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.147604    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"]
Nov 03 11:32:29.148293 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148272    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" containerName="payload-pre-processing"
Nov 03 11:32:29.148371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148297    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" containerName="payload-pre-processing"
Nov 03 11:32:29.148371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148324    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy"
Nov 03 11:32:29.148371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148333    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy"
Nov 03 11:32:29.148371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148367    5817 cpu_manager.go:414] "containerMap: removing container" podUID="2d28be58-89ea-4734-a9a7-f11bbd106db2" containerName="payload-processing"
Nov 03 11:32:29.148522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148374    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d28be58-89ea-4734-a9a7-f11bbd106db2" containerName="payload-processing"
Nov 03 11:32:29.148522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148467    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2aa5d5b-afba-46ef-b7fc-fa9cf57a2472" containerName="payload-pre-processing"
Nov 03 11:32:29.148522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148478    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a65377-d9b1-4c74-9aa1-ec3c80aea824" containerName="istio-proxy"
Nov 03 11:32:29.148522 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.148493    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d28be58-89ea-4734-a9a7-f11bbd106db2" containerName="payload-processing"
Nov 03 11:32:29.153731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.153542    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.161718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.161693    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"]
Nov 03 11:32:29.195938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.195904    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:32:29.196060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.195939    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:32:29.271963 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.271929    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:29.297242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.297218    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrczp\" (UniqueName: \"kubernetes.io/projected/e233912d-c87c-413c-be9d-2a0b1a5ae43c-kube-api-access-hrczp\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.297393 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.297359    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.297453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.297421    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.399040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.398971    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.399167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.399048    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrczp\" (UniqueName: \"kubernetes.io/projected/e233912d-c87c-413c-be9d-2a0b1a5ae43c-kube-api-access-hrczp\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.399167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.399134    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.399690 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.399672    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.401660 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.401644    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.419451 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.419430    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrczp\" (UniqueName: \"kubernetes.io/projected/e233912d-c87c-413c-be9d-2a0b1a5ae43c-kube-api-access-hrczp\") pod \"maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.469592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.469541    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:29.758310 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:32:29.758267    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode233912d_c87c_413c_be9d_2a0b1a5ae43c.slice/crio-c34599cbd3b2ed3c07a2bdf86bc3d1568111b676dac4de720a2018db792618be WatchSource:0}: Error finding container c34599cbd3b2ed3c07a2bdf86bc3d1568111b676dac4de720a2018db792618be: Status 404 returned error can't find the container with id c34599cbd3b2ed3c07a2bdf86bc3d1568111b676dac4de720a2018db792618be
Nov 03 11:32:29.759220 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:29.759190    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"]
Nov 03 11:32:30.099378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:30.099294    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f" event={"ID":"e233912d-c87c-413c-be9d-2a0b1a5ae43c","Type":"ContainerStarted","Data":"c34599cbd3b2ed3c07a2bdf86bc3d1568111b676dac4de720a2018db792618be"}
Nov 03 11:32:30.370694 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:30.370563    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:31.091306 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.091282    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 11:32:31.109092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.109065    5817 generic.go:356] "Generic (PLEG): container finished" podID="bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" containerID="a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513" exitCode=0
Nov 03 11:32:31.109238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.109111    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"
Nov 03 11:32:31.109238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.109158    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp" event={"ID":"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1","Type":"ContainerDied","Data":"a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513"}
Nov 03 11:32:31.109238 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.109222    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp" event={"ID":"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1","Type":"ContainerDied","Data":"b5e4bc283535b11c5a36d543751d691125f33d3e158b1951ae09b49f6a778226"}
Nov 03 11:32:31.109397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.109251    5817 scope.go:122] "RemoveContainer" containerID="a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513"
Nov 03 11:32:31.111499 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.111477    5817 generic.go:356] "Generic (PLEG): container finished" podID="e233912d-c87c-413c-be9d-2a0b1a5ae43c" containerID="f92b288aa2e29e923be0df9ed386970b2ae21cd347ad55e41d5288b5baf9edf1" exitCode=0
Nov 03 11:32:31.111654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.111560    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f" event={"ID":"e233912d-c87c-413c-be9d-2a0b1a5ae43c","Type":"ContainerDied","Data":"f92b288aa2e29e923be0df9ed386970b2ae21cd347ad55e41d5288b5baf9edf1"}
Nov 03 11:32:31.124012 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.123988    5817 scope.go:122] "RemoveContainer" containerID="a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513"
Nov 03 11:32:31.124449 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:32:31.124423    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513\": container with ID starting with a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513 not found: ID does not exist" containerID="a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513"
Nov 03 11:32:31.125908 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.124460    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513"} err="failed to get container status \"a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513\": rpc error: code = NotFound desc = could not find container \"a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513\": container with ID starting with a6118c7d085954be3a7b022c09768105e44e416df1eeea9e09450773f55cf513 not found: ID does not exist"
Nov 03 11:32:31.139809 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.139655    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:31.219836 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.219800    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-kube-api-access-48jh7\" (UniqueName: \"kubernetes.io/projected/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-kube-api-access-48jh7\") pod \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\" (UID: \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\") "
Nov 03 11:32:31.220051 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.220032    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-plugins-config-volume\") pod \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\" (UID: \"bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1\") "
Nov 03 11:32:31.220373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.220351    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-plugins-config-volume" pod "bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" (UID: "bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:31.222111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.222089    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-kube-api-access-48jh7" pod "bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" (UID: "bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1"). InnerVolumeSpecName "kube-api-access-48jh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:31.300299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.300227    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:31.321658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.321629    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:31.321658 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.321653    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48jh7\" (UniqueName: \"kubernetes.io/projected/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1-kube-api-access-48jh7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:31.433385 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.433362    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"]
Nov 03 11:32:31.437910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.437887    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6012b5-5bcd5d5d55-n97wp"]
Nov 03 11:32:31.780255 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.780224    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:31.910697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.910658    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" path="/var/lib/kubelet/pods/bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1/volumes"
Nov 03 11:32:31.912646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:31.912620    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:32.010869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.010835    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"]
Nov 03 11:32:32.015872 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.015849    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cc8214-qjffs"]
Nov 03 11:32:32.067734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.067654    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:32.202405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.202373    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:32.278759 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.278734    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:32.332998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.332917    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-openshift-ca\") pod \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") "
Nov 03 11:32:32.332998 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.332974    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-xks-ca\") pod \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") "
Nov 03 11:32:32.333245 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.333014    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e233912d-c87c-413c-be9d-2a0b1a5ae43c-kube-api-access-hrczp\" (UniqueName: \"kubernetes.io/projected/e233912d-c87c-413c-be9d-2a0b1a5ae43c-kube-api-access-hrczp\") pod \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\" (UID: \"e233912d-c87c-413c-be9d-2a0b1a5ae43c\") "
Nov 03 11:32:32.333315 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.333275    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-openshift-ca" pod "e233912d-c87c-413c-be9d-2a0b1a5ae43c" (UID: "e233912d-c87c-413c-be9d-2a0b1a5ae43c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:32.333411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.333376    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:32.335373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.335337    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-xks-ca" pod "e233912d-c87c-413c-be9d-2a0b1a5ae43c" (UID: "e233912d-c87c-413c-be9d-2a0b1a5ae43c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:32.335373 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.335365    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e233912d-c87c-413c-be9d-2a0b1a5ae43c-kube-api-access-hrczp" pod "e233912d-c87c-413c-be9d-2a0b1a5ae43c" (UID: "e233912d-c87c-413c-be9d-2a0b1a5ae43c"). InnerVolumeSpecName "kube-api-access-hrczp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:32.434739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.434709    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e233912d-c87c-413c-be9d-2a0b1a5ae43c-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:32.434739 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.434736    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrczp\" (UniqueName: \"kubernetes.io/projected/e233912d-c87c-413c-be9d-2a0b1a5ae43c-kube-api-access-hrczp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:32.541134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.541102    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:32.963501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:32.963453    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:33.131176 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:33.131151    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"
Nov 03 11:32:33.131365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:33.131188    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f" event={"ID":"e233912d-c87c-413c-be9d-2a0b1a5ae43c","Type":"ContainerDied","Data":"c34599cbd3b2ed3c07a2bdf86bc3d1568111b676dac4de720a2018db792618be"}
Nov 03 11:32:33.131365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:33.131247    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c34599cbd3b2ed3c07a2bdf86bc3d1568111b676dac4de720a2018db792618be"
Nov 03 11:32:33.175181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:33.175106    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v" podUID="e5f470b0-c9e6-442d-922f-28562581b49f" containerName="payload-processing" containerID="cri-o://e601c729431cb74112ecc4147072e6cee0cefa3050c54bb0ed4dccc7820048a5" gracePeriod=25
Nov 03 11:32:33.769109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:33.769068    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:33.903192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:33.903146    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a101e7f9-0f65-471a-bdcb-1d70fad062d9" path="/var/lib/kubelet/pods/a101e7f9-0f65-471a-bdcb-1d70fad062d9/volumes"
Nov 03 11:32:34.505441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:34.504777    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:35.166801 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:35.166765    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:35.882700 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:35.882662    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:36.104674 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:36.104641    5817 scope.go:122] "RemoveContainer" containerID="4c5bc212f8e8da7f1fdb9e5b0bd02fe354432a974b49e5121db2dabb7c5bea09"
Nov 03 11:32:36.117335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:36.117309    5817 scope.go:122] "RemoveContainer" containerID="2773534247a9f1321c231c5c6ddefc084a28c3fce03f150e1cf0b55650a1ac97"
Nov 03 11:32:36.128214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:36.128195    5817 scope.go:122] "RemoveContainer" containerID="f186c04e28d2e4868fec42ac50f14003ded04848e6397d31391a6e560b43de28"
Nov 03 11:32:36.140662 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:36.140640    5817 scope.go:122] "RemoveContainer" containerID="3ca756f2cde63e3bde97d3b76185689e9017866664a1d0375373b31b4dcbdca9"
Nov 03 11:32:36.531863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:36.531723    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:37.263335 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:37.263305    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:37.394477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:37.394436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:37.937270 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:37.937199    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:38.637525 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:38.637487    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:39.095125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:39.095093    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:39.246932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:39.246888    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:39.645869 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:39.645834    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:39.896993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:39.896900    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:40.238476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:40.238439    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:40.654975 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:40.654888    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v" podUID="e5f470b0-c9e6-442d-922f-28562581b49f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:40.839827 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:40.839778    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:41.458142 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:41.458079    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:42.137642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:42.137422    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:42.297785 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:42.297752    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:32:42.722734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:42.722678    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:42.853630 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:42.853557    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:42.914022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:42.913986    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"]
Nov 03 11:32:42.917155 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:42.917131    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5711cf-skm8f"]
Nov 03 11:32:42.997372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:42.997293    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:43.131274 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:43.131238    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:43.505227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:43.505193    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:43.771849 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:43.771754    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:43.901084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:43.901052    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e233912d-c87c-413c-be9d-2a0b1a5ae43c" path="/var/lib/kubelet/pods/e233912d-c87c-413c-be9d-2a0b1a5ae43c/volumes"
Nov 03 11:32:44.513636 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:44.513597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:45.170637 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:45.170606    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:45.760535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:45.760496    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:46.558117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:46.558071    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:47.261672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:47.261617    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:47.863063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:47.863028    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:48.547419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:48.547371    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:49.253175 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:49.253140    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:49.400646 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:49.400610    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:49.911420 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:49.911389    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:50.021440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.021414    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"]
Nov 03 11:32:50.021892 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.021878    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e233912d-c87c-413c-be9d-2a0b1a5ae43c" containerName="revoke-keys"
Nov 03 11:32:50.021958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.021893    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e233912d-c87c-413c-be9d-2a0b1a5ae43c" containerName="revoke-keys"
Nov 03 11:32:50.021958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.021909    5817 cpu_manager.go:414] "containerMap: removing container" podUID="bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" containerName="payload-processing"
Nov 03 11:32:50.021958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.021914    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" containerName="payload-processing"
Nov 03 11:32:50.022083 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.021985    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e233912d-c87c-413c-be9d-2a0b1a5ae43c" containerName="revoke-keys"
Nov 03 11:32:50.022083 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.021995    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd47ba3-fe9f-4bc1-b4a0-959bd05ec9e1" containerName="payload-processing"
Nov 03 11:32:50.027672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.027652    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.032727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.032704    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:32:50.032993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.032972    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:32:50.041718 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.038969    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"]
Nov 03 11:32:50.054708 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.054675    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:50.145609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.145539    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.145797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.145694    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7nkk\" (UniqueName: \"kubernetes.io/projected/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-kube-api-access-k7nkk\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.145797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.145733    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.206608 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.206553    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:50.246762 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.246730    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.246969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.246811    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7nkk\" (UniqueName: \"kubernetes.io/projected/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-kube-api-access-k7nkk\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.246969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.246860    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.247440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.247415    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.249462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.249441    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.257123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.257095    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7nkk\" (UniqueName: \"kubernetes.io/projected/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-kube-api-access-k7nkk\") pod \"maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.355165 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.355134    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:50.358159 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.358117    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:50.539554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.539312    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:50.654944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.654909    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v" podUID="e5f470b0-c9e6-442d-922f-28562581b49f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:50.667121 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:50.667099    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"]
Nov 03 11:32:50.668476 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:32:50.668451    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc173dd01_16d1_4d13_b6ad_68a1a9ea1398.slice/crio-d6c8bad64442968a248c4f859422402758a442551b45bdd7e1e2d0e25c107352 WatchSource:0}: Error finding container d6c8bad64442968a248c4f859422402758a442551b45bdd7e1e2d0e25c107352: Status 404 returned error can't find the container with id d6c8bad64442968a248c4f859422402758a442551b45bdd7e1e2d0e25c107352
Nov 03 11:32:51.057011 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:51.056970    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:51.317138 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:51.317103    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9" event={"ID":"c173dd01-16d1-4d13-b6ad-68a1a9ea1398","Type":"ContainerStarted","Data":"61e74c269942fa5b146ad8a8d76064bb302fbfcf2cd5d9e78660a03b90c7faa8"}
Nov 03 11:32:51.317680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:51.317156    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9" event={"ID":"c173dd01-16d1-4d13-b6ad-68a1a9ea1398","Type":"ContainerStarted","Data":"d6c8bad64442968a248c4f859422402758a442551b45bdd7e1e2d0e25c107352"}
Nov 03 11:32:51.333940 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:51.333887    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9" podStartSLOduration=1.333874661 podStartE2EDuration="1.333874661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:32:51.331715291 +0000 UTC m=+3801.537297345" watchObservedRunningTime="2026-11-03 11:32:51.333874661 +0000 UTC m=+3801.539456711"
Nov 03 11:32:52.012169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:52.012127    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:52.327645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:52.327531    5817 generic.go:356] "Generic (PLEG): container finished" podID="c173dd01-16d1-4d13-b6ad-68a1a9ea1398" containerID="61e74c269942fa5b146ad8a8d76064bb302fbfcf2cd5d9e78660a03b90c7faa8" exitCode=0
Nov 03 11:32:52.327645 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:52.327625    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9" event={"ID":"c173dd01-16d1-4d13-b6ad-68a1a9ea1398","Type":"ContainerDied","Data":"61e74c269942fa5b146ad8a8d76064bb302fbfcf2cd5d9e78660a03b90c7faa8"}
Nov 03 11:32:52.407206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:52.407173    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:32:52.414805 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:32:52.414773    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:32:52.814825 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:52.814717    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:53.337345 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.337311    5817 generic.go:356] "Generic (PLEG): container finished" podID="e5f470b0-c9e6-442d-922f-28562581b49f" containerID="e601c729431cb74112ecc4147072e6cee0cefa3050c54bb0ed4dccc7820048a5" exitCode=0
Nov 03 11:32:53.337839 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.337731    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v" event={"ID":"e5f470b0-c9e6-442d-922f-28562581b49f","Type":"ContainerDied","Data":"e601c729431cb74112ecc4147072e6cee0cefa3050c54bb0ed4dccc7820048a5"}
Nov 03 11:32:53.449125 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.449060    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 11:32:53.515706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.515684    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:53.577200 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.577167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:53.591743 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.591719    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f470b0-c9e6-442d-922f-28562581b49f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f470b0-c9e6-442d-922f-28562581b49f-plugins-config-volume\") pod \"e5f470b0-c9e6-442d-922f-28562581b49f\" (UID: \"e5f470b0-c9e6-442d-922f-28562581b49f\") "
Nov 03 11:32:53.591907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.591807    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-kube-api-access-k7nkk\" (UniqueName: \"kubernetes.io/projected/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-kube-api-access-k7nkk\") pod \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") "
Nov 03 11:32:53.591907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.591866    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-openshift-ca\") pod \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") "
Nov 03 11:32:53.592050 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.592024    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-xks-ca\") pod \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\" (UID: \"c173dd01-16d1-4d13-b6ad-68a1a9ea1398\") "
Nov 03 11:32:53.592111 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.592059    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f470b0-c9e6-442d-922f-28562581b49f-kube-api-access-bqft6\" (UniqueName: \"kubernetes.io/projected/e5f470b0-c9e6-442d-922f-28562581b49f-kube-api-access-bqft6\") pod \"e5f470b0-c9e6-442d-922f-28562581b49f\" (UID: \"e5f470b0-c9e6-442d-922f-28562581b49f\") "
Nov 03 11:32:53.592528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.592260    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f470b0-c9e6-442d-922f-28562581b49f-plugins-config-volume" pod "e5f470b0-c9e6-442d-922f-28562581b49f" (UID: "e5f470b0-c9e6-442d-922f-28562581b49f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:53.592528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.592284    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-openshift-ca" pod "c173dd01-16d1-4d13-b6ad-68a1a9ea1398" (UID: "c173dd01-16d1-4d13-b6ad-68a1a9ea1398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:53.592528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.592488    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f470b0-c9e6-442d-922f-28562581b49f-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:53.592528 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.592508    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:53.594352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.594328    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-xks-ca" pod "c173dd01-16d1-4d13-b6ad-68a1a9ea1398" (UID: "c173dd01-16d1-4d13-b6ad-68a1a9ea1398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:53.594457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.594407    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f470b0-c9e6-442d-922f-28562581b49f-kube-api-access-bqft6" pod "e5f470b0-c9e6-442d-922f-28562581b49f" (UID: "e5f470b0-c9e6-442d-922f-28562581b49f"). InnerVolumeSpecName "kube-api-access-bqft6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:53.594976 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.594955    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-kube-api-access-k7nkk" pod "c173dd01-16d1-4d13-b6ad-68a1a9ea1398" (UID: "c173dd01-16d1-4d13-b6ad-68a1a9ea1398"). InnerVolumeSpecName "kube-api-access-k7nkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:53.693918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.693889    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7nkk\" (UniqueName: \"kubernetes.io/projected/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-kube-api-access-k7nkk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:53.693918 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.693916    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c173dd01-16d1-4d13-b6ad-68a1a9ea1398-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:53.694110 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:53.693926    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqft6\" (UniqueName: \"kubernetes.io/projected/e5f470b0-c9e6-442d-922f-28562581b49f-kube-api-access-bqft6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.351703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.351670    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"
Nov 03 11:32:54.352330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.351698    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v" event={"ID":"e5f470b0-c9e6-442d-922f-28562581b49f","Type":"ContainerDied","Data":"e644cd46c91bea4d92fcae2bc88669eb746025f23d40d0c0de1a5faed30886d6"}
Nov 03 11:32:54.352330 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.351774    5817 scope.go:122] "RemoveContainer" containerID="e601c729431cb74112ecc4147072e6cee0cefa3050c54bb0ed4dccc7820048a5"
Nov 03 11:32:54.353702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.353531    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9" event={"ID":"c173dd01-16d1-4d13-b6ad-68a1a9ea1398","Type":"ContainerDied","Data":"d6c8bad64442968a248c4f859422402758a442551b45bdd7e1e2d0e25c107352"}
Nov 03 11:32:54.353838 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.353734    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6c8bad64442968a248c4f859422402758a442551b45bdd7e1e2d0e25c107352"
Nov 03 11:32:54.354029 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.354008    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"
Nov 03 11:32:54.366960 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.366938    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:54.372560 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.372539    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"]
Nov 03 11:32:54.379279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.379259    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cc8214-56c49f49f5-r956v"]
Nov 03 11:32:54.930901 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.930863    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"]
Nov 03 11:32:54.979442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:54.979409    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:55.139643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:55.139606    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:55.659894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:55.659858    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:55.901520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:55.901481    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f470b0-c9e6-442d-922f-28562581b49f" path="/var/lib/kubelet/pods/e5f470b0-c9e6-442d-922f-28562581b49f/volumes"
Nov 03 11:32:56.387721 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:56.387659    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:57.059485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:57.059447    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:57.683062 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:57.683022    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:58.345140 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:58.345103    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:59.043407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:59.043374    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:59.665948 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:59.665902    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:32:59.933278 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:32:59.933240    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:00.234143 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:00.234041    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:00.383230 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:00.383200    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:00.591205 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:00.591099    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:01.050083 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:01.050053    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"]
Nov 03 11:33:01.064085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:01.064056    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"]
Nov 03 11:33:01.183973 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:01.183930    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:01.327803 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:01.327671    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:01.959134 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:01.959097    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:02.662045 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:02.662001    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:03.064478 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.064448    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"]
Nov 03 11:33:03.065032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.065012    5817 cpu_manager.go:414] "containerMap: removing container" podUID="c173dd01-16d1-4d13-b6ad-68a1a9ea1398" containerName="revoke-keys"
Nov 03 11:33:03.065151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.065033    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c173dd01-16d1-4d13-b6ad-68a1a9ea1398" containerName="revoke-keys"
Nov 03 11:33:03.065151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.065061    5817 cpu_manager.go:414] "containerMap: removing container" podUID="e5f470b0-c9e6-442d-922f-28562581b49f" containerName="payload-processing"
Nov 03 11:33:03.065151 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.065068    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f470b0-c9e6-442d-922f-28562581b49f" containerName="payload-processing"
Nov 03 11:33:03.065400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.065168    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f470b0-c9e6-442d-922f-28562581b49f" containerName="payload-processing"
Nov 03 11:33:03.065400 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.065194    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="c173dd01-16d1-4d13-b6ad-68a1a9ea1398" containerName="revoke-keys"
Nov 03 11:33:03.070445 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.070424    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.076040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.076016    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:33:03.076364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.076303    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:33:03.076680 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.076659    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"]
Nov 03 11:33:03.198962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.198923    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.198962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.198959    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v24z\" (UniqueName: \"kubernetes.io/projected/b29adfbc-029a-4730-beb4-2f7bb88030d5-kube-api-access-6v24z\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.199174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.199054    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.212745 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.212698    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:03.301401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.300329    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.301401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.300518    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.301401 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.300551    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v24z\" (UniqueName: \"kubernetes.io/projected/b29adfbc-029a-4730-beb4-2f7bb88030d5-kube-api-access-6v24z\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.302498 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.302461    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.315526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.314306    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v24z\" (UniqueName: \"kubernetes.io/projected/b29adfbc-029a-4730-beb4-2f7bb88030d5-kube-api-access-6v24z\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.325462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.325403    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-710f62ef-q49bb\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.354364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.354324    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:03.392077 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.392047    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:03.521855 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.521817    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:03.686096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.686069    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"]
Nov 03 11:33:03.686991 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:33:03.686965    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29adfbc_029a_4730_beb4_2f7bb88030d5.slice/crio-eac69534a6c900bacd296cbaca51602c261a335ed4dfee78243642d1aa82aa7b WatchSource:0}: Error finding container eac69534a6c900bacd296cbaca51602c261a335ed4dfee78243642d1aa82aa7b: Status 404 returned error can't find the container with id eac69534a6c900bacd296cbaca51602c261a335ed4dfee78243642d1aa82aa7b
Nov 03 11:33:03.721980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:03.721941    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:04.050242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:04.050213    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:04.454108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:04.454075    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb" event={"ID":"b29adfbc-029a-4730-beb4-2f7bb88030d5","Type":"ContainerStarted","Data":"6ff9c8cd5eb2663a1e55144efafb26bb0e30f5c4834df161d484529db7b12dff"}
Nov 03 11:33:04.454354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:04.454139    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb" event={"ID":"b29adfbc-029a-4730-beb4-2f7bb88030d5","Type":"ContainerStarted","Data":"eac69534a6c900bacd296cbaca51602c261a335ed4dfee78243642d1aa82aa7b"}
Nov 03 11:33:04.470826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:04.470741    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:04.474826 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:04.474730    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb" podStartSLOduration=1.474705217 podStartE2EDuration="1.474705217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:33:04.472703281 +0000 UTC m=+3814.678285364" watchObservedRunningTime="2026-11-03 11:33:04.474705217 +0000 UTC m=+3814.680287271"
Nov 03 11:33:05.191925 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:05.191885    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:05.464032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:05.463940    5817 generic.go:356] "Generic (PLEG): container finished" podID="b29adfbc-029a-4730-beb4-2f7bb88030d5" containerID="6ff9c8cd5eb2663a1e55144efafb26bb0e30f5c4834df161d484529db7b12dff" exitCode=0
Nov 03 11:33:05.464032 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:05.463986    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb" event={"ID":"b29adfbc-029a-4730-beb4-2f7bb88030d5","Type":"ContainerDied","Data":"6ff9c8cd5eb2663a1e55144efafb26bb0e30f5c4834df161d484529db7b12dff"}
Nov 03 11:33:05.952479 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:05.952433    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:06.050711 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.050674    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442" podUID="0cc6fa9d-166b-489a-bf44-d9f1787d0146" containerName="payload-processing" containerID="cri-o://8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d" gracePeriod=25
Nov 03 11:33:06.064488 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.064438    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx" podUID="da43d24c-95cb-4245-bf73-80269d37b022" containerName="payload-pre-processing" containerID="cri-o://f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516" gracePeriod=25
Nov 03 11:33:06.609898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.609875    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:06.638603 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.638407    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:06.747161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.747068    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-xks-ca\") pod \"b29adfbc-029a-4730-beb4-2f7bb88030d5\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") "
Nov 03 11:33:06.747161 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.747138    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29adfbc-029a-4730-beb4-2f7bb88030d5-kube-api-access-6v24z\" (UniqueName: \"kubernetes.io/projected/b29adfbc-029a-4730-beb4-2f7bb88030d5-kube-api-access-6v24z\") pod \"b29adfbc-029a-4730-beb4-2f7bb88030d5\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") "
Nov 03 11:33:06.747421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.747256    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-openshift-ca\") pod \"b29adfbc-029a-4730-beb4-2f7bb88030d5\" (UID: \"b29adfbc-029a-4730-beb4-2f7bb88030d5\") "
Nov 03 11:33:06.747928 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.747865    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-openshift-ca" pod "b29adfbc-029a-4730-beb4-2f7bb88030d5" (UID: "b29adfbc-029a-4730-beb4-2f7bb88030d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:06.749607 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.749548    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-xks-ca" pod "b29adfbc-029a-4730-beb4-2f7bb88030d5" (UID: "b29adfbc-029a-4730-beb4-2f7bb88030d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:33:06.749828 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.749804    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29adfbc-029a-4730-beb4-2f7bb88030d5-kube-api-access-6v24z" pod "b29adfbc-029a-4730-beb4-2f7bb88030d5" (UID: "b29adfbc-029a-4730-beb4-2f7bb88030d5"). InnerVolumeSpecName "kube-api-access-6v24z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:06.848610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.848554    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:06.848610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.848611    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29adfbc-029a-4730-beb4-2f7bb88030d5-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:06.848778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:06.848622    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v24z\" (UniqueName: \"kubernetes.io/projected/b29adfbc-029a-4730-beb4-2f7bb88030d5-kube-api-access-6v24z\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:07.095389 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:07.095318    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:33:07.271600 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:07.265808    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:07.481367 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:07.481339    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb"
Nov 03 11:33:07.481563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:07.481383    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710f62ef-q49bb" event={"ID":"b29adfbc-029a-4730-beb4-2f7bb88030d5","Type":"ContainerDied","Data":"eac69534a6c900bacd296cbaca51602c261a335ed4dfee78243642d1aa82aa7b"}
Nov 03 11:33:07.481563 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:07.481431    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eac69534a6c900bacd296cbaca51602c261a335ed4dfee78243642d1aa82aa7b"
Nov 03 11:33:07.610639 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:07.610291    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442" podUID="0cc6fa9d-166b-489a-bf44-d9f1787d0146" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:07.614604 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:07.612480    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx" podUID="da43d24c-95cb-4245-bf73-80269d37b022" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:08.065448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:08.065402    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:08.907023 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:08.906987    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:09.105788 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:09.105755    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:09.566642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:09.566597    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:09.952178 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:09.952146    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main" containerID="cri-o://744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854" gracePeriod=45
Nov 03 11:33:10.215861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.215786    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 11:33:10.280914 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.280879    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-tmp-dir\") pod \"5795f519-3787-4f61-b640-6b828a6a56b8\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") "
Nov 03 11:33:10.281114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.280923    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-home\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-home\") pod \"5795f519-3787-4f61-b640-6b828a6a56b8\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") "
Nov 03 11:33:10.281114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.280965    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5795f519-3787-4f61-b640-6b828a6a56b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5795f519-3787-4f61-b640-6b828a6a56b8-tls-certs\") pod \"5795f519-3787-4f61-b640-6b828a6a56b8\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") "
Nov 03 11:33:10.281114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.281007    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-dshm\") pod \"5795f519-3787-4f61-b640-6b828a6a56b8\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") "
Nov 03 11:33:10.281114 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.281082    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5795f519-3787-4f61-b640-6b828a6a56b8-kube-api-access-brkbf\" (UniqueName: \"kubernetes.io/projected/5795f519-3787-4f61-b640-6b828a6a56b8-kube-api-access-brkbf\") pod \"5795f519-3787-4f61-b640-6b828a6a56b8\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") "
Nov 03 11:33:10.281387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.281256    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-tmp-dir" pod "5795f519-3787-4f61-b640-6b828a6a56b8" (UID: "5795f519-3787-4f61-b640-6b828a6a56b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.281597 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.281559    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.281882 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.281858    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-home" pod "5795f519-3787-4f61-b640-6b828a6a56b8" (UID: "5795f519-3787-4f61-b640-6b828a6a56b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.283727 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.283704    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5795f519-3787-4f61-b640-6b828a6a56b8-tls-certs" pod "5795f519-3787-4f61-b640-6b828a6a56b8" (UID: "5795f519-3787-4f61-b640-6b828a6a56b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:33:10.284199 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.284174    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5795f519-3787-4f61-b640-6b828a6a56b8-kube-api-access-brkbf" pod "5795f519-3787-4f61-b640-6b828a6a56b8" (UID: "5795f519-3787-4f61-b640-6b828a6a56b8"). InnerVolumeSpecName "kube-api-access-brkbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:10.284362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.284339    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-dshm" pod "5795f519-3787-4f61-b640-6b828a6a56b8" (UID: "5795f519-3787-4f61-b640-6b828a6a56b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.382081 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.382041    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-model-cache\") pod \"5795f519-3787-4f61-b640-6b828a6a56b8\" (UID: \"5795f519-3787-4f61-b640-6b828a6a56b8\") "
Nov 03 11:33:10.382459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.382363    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-model-cache" pod "5795f519-3787-4f61-b640-6b828a6a56b8" (UID: "5795f519-3787-4f61-b640-6b828a6a56b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.383814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.383776    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.383919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.383818    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5795f519-3787-4f61-b640-6b828a6a56b8-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.383919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.383832    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.383919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.383848    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brkbf\" (UniqueName: \"kubernetes.io/projected/5795f519-3787-4f61-b640-6b828a6a56b8-kube-api-access-brkbf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.383919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.383864    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5795f519-3787-4f61-b640-6b828a6a56b8-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.387360 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.387167    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:10.508632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.508544    5817 generic.go:356] "Generic (PLEG): container finished" podID="5795f519-3787-4f61-b640-6b828a6a56b8" containerID="744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854" exitCode=0
Nov 03 11:33:10.508632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.508614    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"
Nov 03 11:33:10.508632 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.508615    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" event={"ID":"5795f519-3787-4f61-b640-6b828a6a56b8","Type":"ContainerDied","Data":"744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854"}
Nov 03 11:33:10.508876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.508660    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9" event={"ID":"5795f519-3787-4f61-b640-6b828a6a56b8","Type":"ContainerDied","Data":"ed7cba79e1d697c8df64016c13cc8a282dba0f32b0c23a608e80cdfd2fe5d708"}
Nov 03 11:33:10.508876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.508679    5817 scope.go:122] "RemoveContainer" containerID="744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854"
Nov 03 11:33:10.519539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.519520    5817 scope.go:122] "RemoveContainer" containerID="744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854"
Nov 03 11:33:10.519898 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:10.519878    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854\": container with ID starting with 744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854 not found: ID does not exist" containerID="744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854"
Nov 03 11:33:10.519958 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.519906    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854"} err="failed to get container status \"744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854\": rpc error: code = NotFound desc = could not find container \"744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854\": container with ID starting with 744031c131b844b5c0518874f067d904907d06893c48ca4cabe96ccb82c6e854 not found: ID does not exist"
Nov 03 11:33:10.536565 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.536538    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"]
Nov 03 11:33:10.539744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.539722    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-010232/premium-w1-010232-kserve-5c46c98877-v5wz9"]
Nov 03 11:33:10.560204 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.560179    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:10.612851 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.612807    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"]
Nov 03 11:33:10.613072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.613047    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy" containerID="cri-o://fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276" gracePeriod=30
Nov 03 11:33:10.701387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:10.701354    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:11.901008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:11.900972    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" path="/var/lib/kubelet/pods/5795f519-3787-4f61-b640-6b828a6a56b8/volumes"
Nov 03 11:33:12.212016 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:12.211982    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:12.598693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:12.598601    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:12.735771 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:12.735537    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:13.138234 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:13.138204    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:13.492874 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:13.492842    5817 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:33:13.493054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:13.492895    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:33:13.931719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:13.931681    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:14.541182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:14.541148    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:15.157279 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.157243    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:15.873526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.873503    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:33:15.922378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.922276    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:15.950888 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.950849    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-credential-socket\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.950970    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-certs\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951022    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-socket\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951059    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b52aa617-4064-486f-a01d-993eb9ef36e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52aa617-4064-486f-a01d-993eb9ef36e5-istiod-ca-cert\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951093    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-token\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-token\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951149 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951119    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-kube-api-access-89p5c\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-kube-api-access-89p5c\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951193    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-credential-socket" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:15.951328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951214    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-envoy\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951289    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-podinfo\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951318    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-data\") pod \"b52aa617-4064-486f-a01d-993eb9ef36e5\" (UID: \"b52aa617-4064-486f-a01d-993eb9ef36e5\") "
Nov 03 11:33:15.951720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951312    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-certs" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:15.951720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951356    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-socket" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:15.951720 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951562    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-data" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:15.951982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951777    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:15.951982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951798    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:15.951982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951812    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:15.951982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.951826    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:15.952626 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.952536    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52aa617-4064-486f-a01d-993eb9ef36e5-istiod-ca-cert" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:15.953797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.953770    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-token" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:15.954154 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.954118    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-podinfo" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:33:15.954295 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.954264    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-envoy" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:15.954561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:15.954543    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-kube-api-access-89p5c" pod "b52aa617-4064-486f-a01d-993eb9ef36e5" (UID: "b52aa617-4064-486f-a01d-993eb9ef36e5"). InnerVolumeSpecName "kube-api-access-89p5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:16.052377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.052349    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.052377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.052373    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.052377 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.052383    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52aa617-4064-486f-a01d-993eb9ef36e5-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.052703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.052393    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.052703 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.052403    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89p5c\" (UniqueName: \"kubernetes.io/projected/b52aa617-4064-486f-a01d-993eb9ef36e5-kube-api-access-89p5c\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.546702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.546666    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:16.561227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.561196    5817 generic.go:356] "Generic (PLEG): container finished" podID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerID="fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276" exitCode=0
Nov 03 11:33:16.561439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.561271    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"
Nov 03 11:33:16.561439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.561282    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" event={"ID":"b52aa617-4064-486f-a01d-993eb9ef36e5","Type":"ContainerDied","Data":"fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276"}
Nov 03 11:33:16.561439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.561340    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95" event={"ID":"b52aa617-4064-486f-a01d-993eb9ef36e5","Type":"ContainerDied","Data":"5b3aff399a6689c8a4516cd13de0f1ed35eb0e7fcbbd3ebcd2458e19722af020"}
Nov 03 11:33:16.561439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.561369    5817 scope.go:122] "RemoveContainer" containerID="fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276"
Nov 03 11:33:16.575429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.575411    5817 scope.go:122] "RemoveContainer" containerID="fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276"
Nov 03 11:33:16.575752 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:16.575731    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276\": container with ID starting with fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276 not found: ID does not exist" containerID="fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276"
Nov 03 11:33:16.575833 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.575761    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276"} err="failed to get container status \"fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276\": rpc error: code = NotFound desc = could not find container \"fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276\": container with ID starting with fc5a081acb47914b494f5b4b95282e2e63f4d2cbc469a56d99f176242ffdd276 not found: ID does not exist"
Nov 03 11:33:16.586591 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.586551    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"]
Nov 03 11:33:16.591187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:16.591161    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-nhn95"]
Nov 03 11:33:17.254157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:17.254107    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:17.610147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:17.610051    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442" podUID="0cc6fa9d-166b-489a-bf44-d9f1787d0146" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:17.610147 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:17.610087    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx" podUID="da43d24c-95cb-4245-bf73-80269d37b022" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:17.855910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:17.855875    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:17.902893 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:17.902803    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" path="/var/lib/kubelet/pods/b52aa617-4064-486f-a01d-993eb9ef36e5/volumes"
Nov 03 11:33:18.367847 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:18.367818    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:33:18.532239 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:18.532207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:19.165829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:19.165799    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:19.790585 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:19.790534    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:20.275259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:20.275220    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:20.409552 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:20.409519    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:20.504601 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:20.504534    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"]
Nov 03 11:33:20.509046 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:20.509020    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010232-8z9m9"]
Nov 03 11:33:20.543476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:20.543395    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:20.713976 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:20.713939    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:21.059076 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.059029    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:21.155065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155032    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"]
Nov 03 11:33:21.155553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155537    5817 cpu_manager.go:414] "containerMap: removing container" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main"
Nov 03 11:33:21.155553 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155554    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main"
Nov 03 11:33:21.155744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155600    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy"
Nov 03 11:33:21.155744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155606    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy"
Nov 03 11:33:21.155744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155630    5817 cpu_manager.go:414] "containerMap: removing container" podUID="b29adfbc-029a-4730-beb4-2f7bb88030d5" containerName="revoke-keys"
Nov 03 11:33:21.155744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155635    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29adfbc-029a-4730-beb4-2f7bb88030d5" containerName="revoke-keys"
Nov 03 11:33:21.155744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155707    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29adfbc-029a-4730-beb4-2f7bb88030d5" containerName="revoke-keys"
Nov 03 11:33:21.155744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155720    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="5795f519-3787-4f61-b640-6b828a6a56b8" containerName="main"
Nov 03 11:33:21.155744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.155728    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="b52aa617-4064-486f-a01d-993eb9ef36e5" containerName="istio-proxy"
Nov 03 11:33:21.171410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.171376    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"]
Nov 03 11:33:21.171530 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.171483    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.177750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.177728    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:33:21.177900 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.177830    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:33:21.200701 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.200673    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7nfg\" (UniqueName: \"kubernetes.io/projected/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-kube-api-access-g7nfg\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.200852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.200791    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.200852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.200820    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.302418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.302380    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7nfg\" (UniqueName: \"kubernetes.io/projected/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-kube-api-access-g7nfg\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.302654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.302469    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.302654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.302493    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.303352 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.303316    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.304962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.304940    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.310349 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.310292    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7nfg\" (UniqueName: \"kubernetes.io/projected/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-kube-api-access-g7nfg\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.486511 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.486478    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:21.554681 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.552433    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:21.779328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.779296    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"]
Nov 03 11:33:21.780088 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:33:21.780063    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf52d4a_0a95_4e48_9a38_2ae801491f3e.slice/crio-2ad741dabc21c795a372784468cdb05e2897a3c74259a768f0672ae694f78374 WatchSource:0}: Error finding container 2ad741dabc21c795a372784468cdb05e2897a3c74259a768f0672ae694f78374: Status 404 returned error can't find the container with id 2ad741dabc21c795a372784468cdb05e2897a3c74259a768f0672ae694f78374
Nov 03 11:33:21.901117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.901088    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c173dd01-16d1-4d13-b6ad-68a1a9ea1398" path="/var/lib/kubelet/pods/c173dd01-16d1-4d13-b6ad-68a1a9ea1398/volumes"
Nov 03 11:33:21.994379 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:21.994340    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:22.245567 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:22.245515    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:22.391664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:22.389745    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:22.417059 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:22.417022    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:33:22.613407 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:22.613369    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5" event={"ID":"eaf52d4a-0a95-4e48-9a38-2ae801491f3e","Type":"ContainerStarted","Data":"43fe69347960c8715af51439cadd1be969efd3f7282dcb329d3a687627f1bb8a"}
Nov 03 11:33:22.613640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:22.613435    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5" event={"ID":"eaf52d4a-0a95-4e48-9a38-2ae801491f3e","Type":"ContainerStarted","Data":"2ad741dabc21c795a372784468cdb05e2897a3c74259a768f0672ae694f78374"}
Nov 03 11:33:22.632470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:22.632424    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5" podStartSLOduration=1.632407933 podStartE2EDuration="1.632407933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:33:22.632131165 +0000 UTC m=+3832.837713216" watchObservedRunningTime="2026-11-03 11:33:22.632407933 +0000 UTC m=+3832.837989988"
Nov 03 11:33:23.078610 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:23.078481    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:23.640403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:23.640370    5817 generic.go:356] "Generic (PLEG): container finished" podID="eaf52d4a-0a95-4e48-9a38-2ae801491f3e" containerID="43fe69347960c8715af51439cadd1be969efd3f7282dcb329d3a687627f1bb8a" exitCode=0
Nov 03 11:33:23.640822 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:23.640455    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5" event={"ID":"eaf52d4a-0a95-4e48-9a38-2ae801491f3e","Type":"ContainerDied","Data":"43fe69347960c8715af51439cadd1be969efd3f7282dcb329d3a687627f1bb8a"}
Nov 03 11:33:23.796434 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:23.796398    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:24.216453 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.216415    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:24.464724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.464675    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:24.797987 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.797965    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:24.843813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.843787    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-xks-ca\") pod \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") "
Nov 03 11:33:24.843984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.843836    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-kube-api-access-g7nfg\" (UniqueName: \"kubernetes.io/projected/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-kube-api-access-g7nfg\") pod \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") "
Nov 03 11:33:24.843984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.843926    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-openshift-ca\") pod \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\" (UID: \"eaf52d4a-0a95-4e48-9a38-2ae801491f3e\") "
Nov 03 11:33:24.844314 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.844282    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-openshift-ca" pod "eaf52d4a-0a95-4e48-9a38-2ae801491f3e" (UID: "eaf52d4a-0a95-4e48-9a38-2ae801491f3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:24.846705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.846674    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-xks-ca" pod "eaf52d4a-0a95-4e48-9a38-2ae801491f3e" (UID: "eaf52d4a-0a95-4e48-9a38-2ae801491f3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:33:24.846705 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.846689    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-kube-api-access-g7nfg" pod "eaf52d4a-0a95-4e48-9a38-2ae801491f3e" (UID: "eaf52d4a-0a95-4e48-9a38-2ae801491f3e"). InnerVolumeSpecName "kube-api-access-g7nfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:24.944797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.944772    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:24.944797 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.944794    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7nfg\" (UniqueName: \"kubernetes.io/projected/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-kube-api-access-g7nfg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:24.944955 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:24.944806    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf52d4a-0a95-4e48-9a38-2ae801491f3e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:25.020212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:25.020178    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"]
Nov 03 11:33:25.023526 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:25.023497    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9023c992-vkmw8"]
Nov 03 11:33:25.145841 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:25.145791    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:25.665490 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:25.665462    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"
Nov 03 11:33:25.665814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:25.665457    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5" event={"ID":"eaf52d4a-0a95-4e48-9a38-2ae801491f3e","Type":"ContainerDied","Data":"2ad741dabc21c795a372784468cdb05e2897a3c74259a768f0672ae694f78374"}
Nov 03 11:33:25.665814 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:25.665632    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad741dabc21c795a372784468cdb05e2897a3c74259a768f0672ae694f78374"
Nov 03 11:33:25.756644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:25.756605    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:25.901247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:25.901211    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a4e895-c827-4437-9ae9-c1ba8d0bc55e" path="/var/lib/kubelet/pods/77a4e895-c827-4437-9ae9-c1ba8d0bc55e/volumes"
Nov 03 11:33:26.315503 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.315425    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 11:33:26.360457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.360186    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc6fa9d-166b-489a-bf44-d9f1787d0146-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc6fa9d-166b-489a-bf44-d9f1787d0146-plugins-config-volume\") pod \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\" (UID: \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\") "
Nov 03 11:33:26.360457 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.360237    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc6fa9d-166b-489a-bf44-d9f1787d0146-kube-api-access-z7mxf\" (UniqueName: \"kubernetes.io/projected/0cc6fa9d-166b-489a-bf44-d9f1787d0146-kube-api-access-z7mxf\") pod \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\" (UID: \"0cc6fa9d-166b-489a-bf44-d9f1787d0146\") "
Nov 03 11:33:26.361936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.361911    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc6fa9d-166b-489a-bf44-d9f1787d0146-plugins-config-volume" pod "0cc6fa9d-166b-489a-bf44-d9f1787d0146" (UID: "0cc6fa9d-166b-489a-bf44-d9f1787d0146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:26.363071 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.363039    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc6fa9d-166b-489a-bf44-d9f1787d0146-kube-api-access-z7mxf" pod "0cc6fa9d-166b-489a-bf44-d9f1787d0146" (UID: "0cc6fa9d-166b-489a-bf44-d9f1787d0146"). InnerVolumeSpecName "kube-api-access-z7mxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:26.379343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.379323    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 11:33:26.469598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.461055    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da43d24c-95cb-4245-bf73-80269d37b022-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d24c-95cb-4245-bf73-80269d37b022-plugins-config-volume\") pod \"da43d24c-95cb-4245-bf73-80269d37b022\" (UID: \"da43d24c-95cb-4245-bf73-80269d37b022\") "
Nov 03 11:33:26.469598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.461102    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da43d24c-95cb-4245-bf73-80269d37b022-kube-api-access-s26pm\" (UniqueName: \"kubernetes.io/projected/da43d24c-95cb-4245-bf73-80269d37b022-kube-api-access-s26pm\") pod \"da43d24c-95cb-4245-bf73-80269d37b022\" (UID: \"da43d24c-95cb-4245-bf73-80269d37b022\") "
Nov 03 11:33:26.469598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.461970    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:26.469598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.462346    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc6fa9d-166b-489a-bf44-d9f1787d0146-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:26.469598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.462371    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7mxf\" (UniqueName: \"kubernetes.io/projected/0cc6fa9d-166b-489a-bf44-d9f1787d0146-kube-api-access-z7mxf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:26.469598 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.462447    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da43d24c-95cb-4245-bf73-80269d37b022-plugins-config-volume" pod "da43d24c-95cb-4245-bf73-80269d37b022" (UID: "da43d24c-95cb-4245-bf73-80269d37b022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:26.470786 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.470751    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da43d24c-95cb-4245-bf73-80269d37b022-kube-api-access-s26pm" pod "da43d24c-95cb-4245-bf73-80269d37b022" (UID: "da43d24c-95cb-4245-bf73-80269d37b022"). InnerVolumeSpecName "kube-api-access-s26pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:26.563750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.563719    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d24c-95cb-4245-bf73-80269d37b022-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:26.563750 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.563747    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s26pm\" (UniqueName: \"kubernetes.io/projected/da43d24c-95cb-4245-bf73-80269d37b022-kube-api-access-s26pm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:26.676640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.676532    5817 generic.go:356] "Generic (PLEG): container finished" podID="da43d24c-95cb-4245-bf73-80269d37b022" containerID="f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516" exitCode=0
Nov 03 11:33:26.676640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.676621    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"
Nov 03 11:33:26.676640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.676625    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx" event={"ID":"da43d24c-95cb-4245-bf73-80269d37b022","Type":"ContainerDied","Data":"f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516"}
Nov 03 11:33:26.676939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.676688    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx" event={"ID":"da43d24c-95cb-4245-bf73-80269d37b022","Type":"ContainerDied","Data":"b7696950bbf7fbc293693f6b4c3ccd1ff2d1001af12e8475dfaec14b5c5d927e"}
Nov 03 11:33:26.676939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.676719    5817 scope.go:122] "RemoveContainer" containerID="f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516"
Nov 03 11:33:26.678286 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.678267    5817 generic.go:356] "Generic (PLEG): container finished" podID="0cc6fa9d-166b-489a-bf44-d9f1787d0146" containerID="8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d" exitCode=0
Nov 03 11:33:26.678416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.678308    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442" event={"ID":"0cc6fa9d-166b-489a-bf44-d9f1787d0146","Type":"ContainerDied","Data":"8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d"}
Nov 03 11:33:26.678416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.678315    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"
Nov 03 11:33:26.678416 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.678341    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442" event={"ID":"0cc6fa9d-166b-489a-bf44-d9f1787d0146","Type":"ContainerDied","Data":"676fcf78b323a87e572b2dacbb2053a5cf8d2e6e0d982fea11ff6c816a469236"}
Nov 03 11:33:26.689233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.689215    5817 scope.go:122] "RemoveContainer" containerID="f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516"
Nov 03 11:33:26.689525 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:26.689502    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516\": container with ID starting with f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516 not found: ID does not exist" containerID="f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516"
Nov 03 11:33:26.689640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.689537    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516"} err="failed to get container status \"f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516\": rpc error: code = NotFound desc = could not find container \"f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516\": container with ID starting with f1866f47a5695544ffa59699d4af4400bafd493314455077c227781a9ad93516 not found: ID does not exist"
Nov 03 11:33:26.689640 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.689562    5817 scope.go:122] "RemoveContainer" containerID="8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d"
Nov 03 11:33:26.700058 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.700040    5817 scope.go:122] "RemoveContainer" containerID="8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d"
Nov 03 11:33:26.700312 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:26.700295    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d\": container with ID starting with 8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d not found: ID does not exist" containerID="8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d"
Nov 03 11:33:26.700365 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.700320    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d"} err="failed to get container status \"8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d\": rpc error: code = NotFound desc = could not find container \"8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d\": container with ID starting with 8f424b6af7008e7ea11e75d08fefb2609865ff84c089441515ad0e22aff1ed2d not found: ID does not exist"
Nov 03 11:33:26.706052 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.706032    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"]
Nov 03 11:33:26.710769 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.710750    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ce5555-76c84548c9-sn442"]
Nov 03 11:33:26.722446 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.722418    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"]
Nov 03 11:33:26.725039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:26.725017    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ce5555-7967f55dbc-pxlhx"]
Nov 03 11:33:27.149098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.149061    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:27.383236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.383143    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:27.584566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.584472    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:27.742101 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.741954    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"]
Nov 03 11:33:27.742322 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.742163    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy" containerID="cri-o://560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4" gracePeriod=30
Nov 03 11:33:27.827390 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.827353    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:27.900540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.900503    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc6fa9d-166b-489a-bf44-d9f1787d0146" path="/var/lib/kubelet/pods/0cc6fa9d-166b-489a-bf44-d9f1787d0146/volumes"
Nov 03 11:33:27.900966 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.900952    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da43d24c-95cb-4245-bf73-80269d37b022" path="/var/lib/kubelet/pods/da43d24c-95cb-4245-bf73-80269d37b022/volumes"
Nov 03 11:33:27.971920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:27.971883    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:28.649949 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:28.649921    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:29.341773 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:29.341741    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:29.877894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:29.877859    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:33:29.953993 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:29.953947    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:30.683441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:30.683392    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:31.253242 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:31.253207    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:31.283202 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:31.283177    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"]
Nov 03 11:33:31.293562 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:31.293532    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"]
Nov 03 11:33:31.407486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:31.407448    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:31.550181 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:31.550094    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:32.043164 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:32.043131    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:32.743935 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:32.743898    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:32.998256 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:32.998199    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 11:33:33.136239 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136213    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istiod-ca-cert\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.136417 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136291    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-certs\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.136486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136460    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-envoy\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.136547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136514    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-credential-socket\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.136641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136554    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-socket\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.136641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136592    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-certs" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:33.136641 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136611    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istiod-ca-cert" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:33.136815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136653    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-token\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-token\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.136815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136685    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-kube-api-access-dmdqc\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-kube-api-access-dmdqc\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.136815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136744    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-data\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.136815 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136772    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-credential-socket" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:33.137040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136832    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-podinfo\") pod \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\" (UID: \"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec\") "
Nov 03 11:33:33.137040 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.136857    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-socket" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:33.137169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.137107    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-data" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:33.137358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.137305    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.137419 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.137382    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.137470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.137444    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.137529 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.137474    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.137529 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.137488    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.138944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.138913    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-envoy" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:33.139422 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.139395    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-podinfo" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:33:33.139520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.139465    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-token" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:33.139520 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.139497    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-kube-api-access-dmdqc" pod "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" (UID: "fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec"). InnerVolumeSpecName "kube-api-access-dmdqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:33.238702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.238666    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.238702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.238696    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.238702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.238708    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.238950 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.238720    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdqc\" (UniqueName: \"kubernetes.io/projected/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec-kube-api-access-dmdqc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:33.468961 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.468925    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:33.625501 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.625440    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:33.663369 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.663342    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"]
Nov 03 11:33:33.672808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.672782    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-836589-lrrr5"]
Nov 03 11:33:33.755936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.755843    5817 generic.go:356] "Generic (PLEG): container finished" podID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerID="560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4" exitCode=0
Nov 03 11:33:33.755936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.755903    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"
Nov 03 11:33:33.756588 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.755933    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" event={"ID":"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec","Type":"ContainerDied","Data":"560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4"}
Nov 03 11:33:33.756588 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.755997    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w" event={"ID":"fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec","Type":"ContainerDied","Data":"74f0131c2af8cb306d57468f0d5d967eb2c3807380fd2562a395f447d253e613"}
Nov 03 11:33:33.756588 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.756023    5817 scope.go:122] "RemoveContainer" containerID="560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4"
Nov 03 11:33:33.768842 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.768820    5817 scope.go:122] "RemoveContainer" containerID="560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4"
Nov 03 11:33:33.769175 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:33.769153    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4\": container with ID starting with 560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4 not found: ID does not exist" containerID="560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4"
Nov 03 11:33:33.769269 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.769183    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4"} err="failed to get container status \"560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4\": rpc error: code = NotFound desc = could not find container \"560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4\": container with ID starting with 560442c3aca71a29ad0ede9a6d1d3b27a50120b9555d81841c6b446a11a369d4 not found: ID does not exist"
Nov 03 11:33:33.772276 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.772255    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:33.781653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.781557    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"]
Nov 03 11:33:33.785755 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.785734    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-5pm2w"]
Nov 03 11:33:33.901053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.901016    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf52d4a-0a95-4e48-9a38-2ae801491f3e" path="/var/lib/kubelet/pods/eaf52d4a-0a95-4e48-9a38-2ae801491f3e/volumes"
Nov 03 11:33:33.901473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.901458    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" path="/var/lib/kubelet/pods/fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec/volumes"
Nov 03 11:33:33.907602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:33.907555    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:34.171599 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:34.171496    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:34.308858 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:34.308817    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:34.683074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:34.683037    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:35.044549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:35.044464    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"]
Nov 03 11:33:35.052339 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:35.052309    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54aad279-fpm7j"]
Nov 03 11:33:35.398343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:35.398244    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:35.903009 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:35.902971    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c95484-2e1e-4567-b5a6-73da33e4ca1f" path="/var/lib/kubelet/pods/49c95484-2e1e-4567-b5a6-73da33e4ca1f/volumes"
Nov 03 11:33:36.110233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:36.110169    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:36.273447 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:36.273416    5817 scope.go:122] "RemoveContainer" containerID="876e631a8f801aec0a26b7bab3f5fb72cb110890ac40630a9f97b6c257dbe9a6"
Nov 03 11:33:36.283672 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:36.283638    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t" podUID="f4d72517-c9c3-4c7c-a75d-86b42d6c644f" containerName="payload-processing" containerID="cri-o://dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa" gracePeriod=25
Nov 03 11:33:36.285098 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:36.285069    5817 scope.go:122] "RemoveContainer" containerID="01e52de53752543ee0aa40484822f8b908465051f5b82d7f39bdd0a9f98cd0c3"
Nov 03 11:33:36.294157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:36.294127    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl" podUID="7155fb21-319e-47f0-8680-359ca1a613b0" containerName="payload-pre-processing" containerID="cri-o://1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a" gracePeriod=25
Nov 03 11:33:36.869320 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:36.869286    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:37.653644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:37.653606    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:37.939074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:37.939038    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl" podUID="7155fb21-319e-47f0-8680-359ca1a613b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:38.370861 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:38.370780    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:38.614294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:38.614261    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:33:38.948596 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:38.948538    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t" podUID="f4d72517-c9c3-4c7c-a75d-86b42d6c644f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:39.086363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:39.086324    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:39.755409 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:39.755377    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:40.652292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:40.652252    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:40.917049 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:40.916958    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:41.078937 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:41.078879    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:41.219464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:41.219428    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:41.352820 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:41.352783    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"]
Nov 03 11:33:41.353063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:41.353035    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy" containerID="cri-o://e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9" gracePeriod=30
Nov 03 11:33:41.437459 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:41.437176    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:41.774905 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:41.774856    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:41.940765 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:41.940720    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:42.716358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:42.716322    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:43.509734 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:43.509701    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:44.115919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:44.115880    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:45.552903 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:45.552865    5817 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:33:45.553378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:45.552931    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:33:45.802075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:45.802038    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:46.185171 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.185136    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:46.606760 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.606730    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:33:46.777790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.777705    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-envoy\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.777980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.777806    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-credential-socket\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.777980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.777871    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-socket\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.777980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.777939    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-podinfo\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.777980 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.777967    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-data\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.778214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.777995    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-kube-api-access-vlc7m\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-kube-api-access-vlc7m\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.778214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778029    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-token\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-token\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.778214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778067    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f820a885-2323-41ca-aaa5-0f55ccd76134-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f820a885-2323-41ca-aaa5-0f55ccd76134-istiod-ca-cert\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.778214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778109    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-socket" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.778214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778146    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-certs\") pod \"f820a885-2323-41ca-aaa5-0f55ccd76134\" (UID: \"f820a885-2323-41ca-aaa5-0f55ccd76134\") "
Nov 03 11:33:46.778214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778129    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-credential-socket" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.778214 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778200    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-data" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.778696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778531    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.778696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778563    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.778696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778596    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.778696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778529    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-certs" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.778696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.778553    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f820a885-2323-41ca-aaa5-0f55ccd76134-istiod-ca-cert" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:46.780486 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.780464    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-kube-api-access-vlc7m" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "kube-api-access-vlc7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:46.780592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.780494    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-token" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:46.780829 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.780793    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-podinfo" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:33:46.780984 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.780841    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-envoy" pod "f820a885-2323-41ca-aaa5-0f55ccd76134" (UID: "f820a885-2323-41ca-aaa5-0f55ccd76134"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.856354 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.856325    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"]
Nov 03 11:33:46.868364 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.868337    5817 generic.go:356] "Generic (PLEG): container finished" podID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerID="e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9" exitCode=0
Nov 03 11:33:46.868464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.868381    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"
Nov 03 11:33:46.868464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.868420    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" event={"ID":"f820a885-2323-41ca-aaa5-0f55ccd76134","Type":"ContainerDied","Data":"e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9"}
Nov 03 11:33:46.868618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.868471    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb" event={"ID":"f820a885-2323-41ca-aaa5-0f55ccd76134","Type":"ContainerDied","Data":"7292e463efbbaf2157f8a3f703ae10d9d00fe4122cf7c5e57ca31ac3ac00ec42"}
Nov 03 11:33:46.868618 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.868493    5817 scope.go:122] "RemoveContainer" containerID="e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9"
Nov 03 11:33:46.879642 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.879616    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.879761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.879644    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlc7m\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-kube-api-access-vlc7m\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.879761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.879661    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.879761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.879674    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f820a885-2323-41ca-aaa5-0f55ccd76134-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.879761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.879688    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.879761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.879701    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f820a885-2323-41ca-aaa5-0f55ccd76134-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.880358 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.880346    5817 scope.go:122] "RemoveContainer" containerID="e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9"
Nov 03 11:33:46.880638 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:46.880620    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9\": container with ID starting with e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9 not found: ID does not exist" containerID="e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9"
Nov 03 11:33:46.880693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.880645    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9"} err="failed to get container status \"e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9\": rpc error: code = NotFound desc = could not find container \"e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9\": container with ID starting with e893129d65181bd90aeb0f3e31c5bb939d2e7b6a62f67a4309217d58968886f9 not found: ID does not exist"
Nov 03 11:33:46.912738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.912714    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"]
Nov 03 11:33:46.920084 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.920062    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:46.920259 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:46.920242    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-gt6pb"]
Nov 03 11:33:47.068232 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:47.068114    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:47.650355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:47.650317    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:47.899790 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:47.899757    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" path="/var/lib/kubelet/pods/f820a885-2323-41ca-aaa5-0f55ccd76134/volumes"
Nov 03 11:33:47.938544 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:47.938514    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl" podUID="7155fb21-319e-47f0-8680-359ca1a613b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:48.352810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:48.352707    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:48.949355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:48.949314    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t" podUID="f4d72517-c9c3-4c7c-a75d-86b42d6c644f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:49.063212 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:49.063152    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:49.743919 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:49.743882    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:50.354929 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:50.354896    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:50.960449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:50.960414    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:51.562133 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:51.562088    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:52.218906 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:52.218862    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"]
Nov 03 11:33:52.242517 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:52.242484    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"]
Nov 03 11:33:52.256157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:52.256119    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:52.403547 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:52.403513    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:52.412488 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:52.412459    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:33:52.953469 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:52.953436    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:53.394476 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394410    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"]
Nov 03 11:33:53.394868 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394854    5817 cpu_manager.go:414] "containerMap: removing container" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy"
Nov 03 11:33:53.394920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394873    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy"
Nov 03 11:33:53.394920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394904    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy"
Nov 03 11:33:53.394920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394909    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy"
Nov 03 11:33:53.394920 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394918    5817 cpu_manager.go:414] "containerMap: removing container" podUID="da43d24c-95cb-4245-bf73-80269d37b022" containerName="payload-pre-processing"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394924    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da43d24c-95cb-4245-bf73-80269d37b022" containerName="payload-pre-processing"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394932    5817 cpu_manager.go:414] "containerMap: removing container" podUID="0cc6fa9d-166b-489a-bf44-d9f1787d0146" containerName="payload-processing"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394938    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc6fa9d-166b-489a-bf44-d9f1787d0146" containerName="payload-processing"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394957    5817 cpu_manager.go:414] "containerMap: removing container" podUID="eaf52d4a-0a95-4e48-9a38-2ae801491f3e" containerName="revoke-keys"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.394963    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf52d4a-0a95-4e48-9a38-2ae801491f3e" containerName="revoke-keys"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.395029    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdda72df-d1e1-4ab8-9665-8bbcb2f2f9ec" containerName="istio-proxy"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.395046    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f820a885-2323-41ca-aaa5-0f55ccd76134" containerName="istio-proxy"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.395052    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="da43d24c-95cb-4245-bf73-80269d37b022" containerName="payload-pre-processing"
Nov 03 11:33:53.395060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.395058    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc6fa9d-166b-489a-bf44-d9f1787d0146" containerName="payload-processing"
Nov 03 11:33:53.395359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.395065    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf52d4a-0a95-4e48-9a38-2ae801491f3e" containerName="revoke-keys"
Nov 03 11:33:53.399763 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.399742    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.403439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.403340    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:33:53.403439 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.403392    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:33:53.406590 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.406552    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"]
Nov 03 11:33:53.439391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.439365    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.439532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.439400    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7tb\" (UniqueName: \"kubernetes.io/projected/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-kube-api-access-qp7tb\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.439532 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.439448    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.540473 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.540441    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.540628 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.540553    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.540714 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.540626    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7tb\" (UniqueName: \"kubernetes.io/projected/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-kube-api-access-qp7tb\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.541192 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.541173    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.542932 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.542911    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.549292 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.549275    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7tb\" (UniqueName: \"kubernetes.io/projected/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-kube-api-access-qp7tb\") pod \"maas-api-revoke-sub-e71f2799-w7rcf\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.715985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.715952    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:53.996619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:53.996590    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"]
Nov 03 11:33:53.998056 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:33:53.998023    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865effdb_6bf7_47ef_830a_b3c4ccdb4ea0.slice/crio-a37b56feea40a15badf9792ac8592d0804af399199f69ac59de41f0f1fc1999c WatchSource:0}: Error finding container a37b56feea40a15badf9792ac8592d0804af399199f69ac59de41f0f1fc1999c: Status 404 returned error can't find the container with id a37b56feea40a15badf9792ac8592d0804af399199f69ac59de41f0f1fc1999c
Nov 03 11:33:54.933169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:54.933135    5817 generic.go:356] "Generic (PLEG): container finished" podID="865effdb-6bf7-47ef-830a-b3c4ccdb4ea0" containerID="ce667256b08c4fe2f191d998b2288ad2f7443db202ef8a16aa12fd4e00c695a4" exitCode=0
Nov 03 11:33:54.933355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:54.933192    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf" event={"ID":"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0","Type":"ContainerDied","Data":"ce667256b08c4fe2f191d998b2288ad2f7443db202ef8a16aa12fd4e00c695a4"}
Nov 03 11:33:54.933355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:54.933243    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf" event={"ID":"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0","Type":"ContainerStarted","Data":"a37b56feea40a15badf9792ac8592d0804af399199f69ac59de41f0f1fc1999c"}
Nov 03 11:33:55.036005 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:55.035940    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:33:56.080923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.080901    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:56.163653 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.163626    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-openshift-ca\") pod \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") "
Nov 03 11:33:56.163845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.163754    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-kube-api-access-qp7tb\" (UniqueName: \"kubernetes.io/projected/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-kube-api-access-qp7tb\") pod \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") "
Nov 03 11:33:56.163845 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.163794    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-xks-ca\") pod \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\" (UID: \"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0\") "
Nov 03 11:33:56.163983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.163957    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-openshift-ca" pod "865effdb-6bf7-47ef-830a-b3c4ccdb4ea0" (UID: "865effdb-6bf7-47ef-830a-b3c4ccdb4ea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:56.164156 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.164138    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:56.166072 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.166049    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-xks-ca" pod "865effdb-6bf7-47ef-830a-b3c4ccdb4ea0" (UID: "865effdb-6bf7-47ef-830a-b3c4ccdb4ea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:33:56.166187 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.166104    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-kube-api-access-qp7tb" pod "865effdb-6bf7-47ef-830a-b3c4ccdb4ea0" (UID: "865effdb-6bf7-47ef-830a-b3c4ccdb4ea0"). InnerVolumeSpecName "kube-api-access-qp7tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:56.265321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.265254    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7tb\" (UniqueName: \"kubernetes.io/projected/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-kube-api-access-qp7tb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:56.265321 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.265291    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865effdb-6bf7-47ef-830a-b3c4ccdb4ea0-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:56.543651 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.543607    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 11:33:56.567397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.567370    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-kube-api-access-l9tfv\" (UniqueName: \"kubernetes.io/projected/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-kube-api-access-l9tfv\") pod \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\" (UID: \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\") "
Nov 03 11:33:56.567554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.567459    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-plugins-config-volume\") pod \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\" (UID: \"f4d72517-c9c3-4c7c-a75d-86b42d6c644f\") "
Nov 03 11:33:56.567985 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.567952    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-plugins-config-volume" pod "f4d72517-c9c3-4c7c-a75d-86b42d6c644f" (UID: "f4d72517-c9c3-4c7c-a75d-86b42d6c644f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:56.570717 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.570692    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-kube-api-access-l9tfv" pod "f4d72517-c9c3-4c7c-a75d-86b42d6c644f" (UID: "f4d72517-c9c3-4c7c-a75d-86b42d6c644f"). InnerVolumeSpecName "kube-api-access-l9tfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:56.573696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.573678    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 11:33:56.668510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.668480    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7155fb21-319e-47f0-8680-359ca1a613b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7155fb21-319e-47f0-8680-359ca1a613b0-plugins-config-volume\") pod \"7155fb21-319e-47f0-8680-359ca1a613b0\" (UID: \"7155fb21-319e-47f0-8680-359ca1a613b0\") "
Nov 03 11:33:56.668693 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.668520    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7155fb21-319e-47f0-8680-359ca1a613b0-kube-api-access-cwzdt\" (UniqueName: \"kubernetes.io/projected/7155fb21-319e-47f0-8680-359ca1a613b0-kube-api-access-cwzdt\") pod \"7155fb21-319e-47f0-8680-359ca1a613b0\" (UID: \"7155fb21-319e-47f0-8680-359ca1a613b0\") "
Nov 03 11:33:56.668897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.668876    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9tfv\" (UniqueName: \"kubernetes.io/projected/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-kube-api-access-l9tfv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:56.668939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.668905    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d72517-c9c3-4c7c-a75d-86b42d6c644f-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:56.668939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.668919    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7155fb21-319e-47f0-8680-359ca1a613b0-plugins-config-volume" pod "7155fb21-319e-47f0-8680-359ca1a613b0" (UID: "7155fb21-319e-47f0-8680-359ca1a613b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:56.670744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.670719    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7155fb21-319e-47f0-8680-359ca1a613b0-kube-api-access-cwzdt" pod "7155fb21-319e-47f0-8680-359ca1a613b0" (UID: "7155fb21-319e-47f0-8680-359ca1a613b0"). InnerVolumeSpecName "kube-api-access-cwzdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:56.770425 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.770396    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7155fb21-319e-47f0-8680-359ca1a613b0-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:56.770425 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.770425    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwzdt\" (UniqueName: \"kubernetes.io/projected/7155fb21-319e-47f0-8680-359ca1a613b0-kube-api-access-cwzdt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:56.955426 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.955393    5817 generic.go:356] "Generic (PLEG): container finished" podID="f4d72517-c9c3-4c7c-a75d-86b42d6c644f" containerID="dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa" exitCode=0
Nov 03 11:33:56.955675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.955465    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"
Nov 03 11:33:56.955675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.955471    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t" event={"ID":"f4d72517-c9c3-4c7c-a75d-86b42d6c644f","Type":"ContainerDied","Data":"dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa"}
Nov 03 11:33:56.955675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.955624    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t" event={"ID":"f4d72517-c9c3-4c7c-a75d-86b42d6c644f","Type":"ContainerDied","Data":"8c6d8c602f18114bd9c27c9deab9ed55603d6d68c94e18824a1e728fedae64fa"}
Nov 03 11:33:56.955675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.955663    5817 scope.go:122] "RemoveContainer" containerID="dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa"
Nov 03 11:33:56.957094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.957032    5817 generic.go:356] "Generic (PLEG): container finished" podID="7155fb21-319e-47f0-8680-359ca1a613b0" containerID="1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a" exitCode=0
Nov 03 11:33:56.957172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.957093    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"
Nov 03 11:33:56.957172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.957117    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl" event={"ID":"7155fb21-319e-47f0-8680-359ca1a613b0","Type":"ContainerDied","Data":"1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a"}
Nov 03 11:33:56.957172 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.957166    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl" event={"ID":"7155fb21-319e-47f0-8680-359ca1a613b0","Type":"ContainerDied","Data":"4a928db28580989f07e6d4f30fc35e2f5209a91a3e0b79a5f1a70cc0cbcf26c0"}
Nov 03 11:33:56.958831 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.958796    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf"
Nov 03 11:33:56.958938 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.958874    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2799-w7rcf" event={"ID":"865effdb-6bf7-47ef-830a-b3c4ccdb4ea0","Type":"ContainerDied","Data":"a37b56feea40a15badf9792ac8592d0804af399199f69ac59de41f0f1fc1999c"}
Nov 03 11:33:56.959055 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.958940    5817 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a37b56feea40a15badf9792ac8592d0804af399199f69ac59de41f0f1fc1999c"
Nov 03 11:33:56.968366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.968347    5817 scope.go:122] "RemoveContainer" containerID="dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa"
Nov 03 11:33:56.968699 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:56.968681    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa\": container with ID starting with dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa not found: ID does not exist" containerID="dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa"
Nov 03 11:33:56.968799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.968706    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa"} err="failed to get container status \"dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa\": rpc error: code = NotFound desc = could not find container \"dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa\": container with ID starting with dd69803e62ee121df142d54dfb5600d2937ef11a0394de01e4fae845e5e74dfa not found: ID does not exist"
Nov 03 11:33:56.968799 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.968721    5817 scope.go:122] "RemoveContainer" containerID="1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a"
Nov 03 11:33:56.978296 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.978278    5817 scope.go:122] "RemoveContainer" containerID="1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a"
Nov 03 11:33:56.978567 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:33:56.978547    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a\": container with ID starting with 1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a not found: ID does not exist" containerID="1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a"
Nov 03 11:33:56.978675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.978601    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a"} err="failed to get container status \"1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a\": rpc error: code = NotFound desc = could not find container \"1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a\": container with ID starting with 1c6ed553836ee9b9cdb223a9128a1d02b38579b7179822bde81b2e9e19bd526a not found: ID does not exist"
Nov 03 11:33:56.989643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.989623    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"]
Nov 03 11:33:56.990173 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:56.990154    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-836589-7589f8657f-6d56t"]
Nov 03 11:33:57.002970 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:57.002936    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"]
Nov 03 11:33:57.008194 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:57.008175    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-836589-ff48b7b7f-z7vwl"]
Nov 03 11:33:57.219169 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:57.219075    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv" podUID="6b827954-c908-4603-998d-501f19de5a7a" containerName="payload-processing" containerID="cri-o://a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e" gracePeriod=25
Nov 03 11:33:57.243682 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:57.243637    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9" podUID="f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" containerName="payload-pre-processing" containerID="cri-o://dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9" gracePeriod=25
Nov 03 11:33:57.906236 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:57.906200    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7155fb21-319e-47f0-8680-359ca1a613b0" path="/var/lib/kubelet/pods/7155fb21-319e-47f0-8680-359ca1a613b0/volumes"
Nov 03 11:33:57.906664 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:57.906624    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d72517-c9c3-4c7c-a75d-86b42d6c644f" path="/var/lib/kubelet/pods/f4d72517-c9c3-4c7c-a75d-86b42d6c644f/volumes"
Nov 03 11:33:59.592449 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:59.592410    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv" podUID="6b827954-c908-4603-998d-501f19de5a7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:33:59.592923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:33:59.592668    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9" podUID="f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:34:01.878748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:01.878708    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" containerID="cri-o://9679a69ef07a09ee16b0884e36dbbd3c4ed8968703d0cc677742354244738756" gracePeriod=45
Nov 03 11:34:01.998092 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:01.998059    5817 generic.go:356] "Generic (PLEG): container finished" podID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerID="9679a69ef07a09ee16b0884e36dbbd3c4ed8968703d0cc677742354244738756" exitCode=0
Nov 03 11:34:01.998218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:01.998115    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" event={"ID":"29455243-b7f4-4059-8066-ddf8e33d6abd","Type":"ContainerDied","Data":"9679a69ef07a09ee16b0884e36dbbd3c4ed8968703d0cc677742354244738756"}
Nov 03 11:34:02.031044 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.031015    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"]
Nov 03 11:34:02.035588 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.035538    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcabd2-7w7sx"]
Nov 03 11:34:02.124454 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.124425    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 11:34:02.227735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.227702    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29455243-b7f4-4059-8066-ddf8e33d6abd-tls-certs\" (UniqueName: \"kubernetes.io/secret/29455243-b7f4-4059-8066-ddf8e33d6abd-tls-certs\") pod \"29455243-b7f4-4059-8066-ddf8e33d6abd\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") "
Nov 03 11:34:02.227945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.227753    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-tmp-dir\") pod \"29455243-b7f4-4059-8066-ddf8e33d6abd\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") "
Nov 03 11:34:02.227945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.227847    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-dshm\") pod \"29455243-b7f4-4059-8066-ddf8e33d6abd\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") "
Nov 03 11:34:02.227945 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.227892    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-home\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-home\") pod \"29455243-b7f4-4059-8066-ddf8e33d6abd\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") "
Nov 03 11:34:02.228441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.227960    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29455243-b7f4-4059-8066-ddf8e33d6abd-kube-api-access-bt5cf\" (UniqueName: \"kubernetes.io/projected/29455243-b7f4-4059-8066-ddf8e33d6abd-kube-api-access-bt5cf\") pod \"29455243-b7f4-4059-8066-ddf8e33d6abd\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") "
Nov 03 11:34:02.228441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.228025    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-tmp-dir" pod "29455243-b7f4-4059-8066-ddf8e33d6abd" (UID: "29455243-b7f4-4059-8066-ddf8e33d6abd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.228441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.228033    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-model-cache\") pod \"29455243-b7f4-4059-8066-ddf8e33d6abd\" (UID: \"29455243-b7f4-4059-8066-ddf8e33d6abd\") "
Nov 03 11:34:02.228441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.228247    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-model-cache" pod "29455243-b7f4-4059-8066-ddf8e33d6abd" (UID: "29455243-b7f4-4059-8066-ddf8e33d6abd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.228441 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.228336    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-home" pod "29455243-b7f4-4059-8066-ddf8e33d6abd" (UID: "29455243-b7f4-4059-8066-ddf8e33d6abd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.228899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.228609    5817 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.228899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.228628    5817 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.228899 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.228639    5817 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.230715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.230679    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29455243-b7f4-4059-8066-ddf8e33d6abd-kube-api-access-bt5cf" pod "29455243-b7f4-4059-8066-ddf8e33d6abd" (UID: "29455243-b7f4-4059-8066-ddf8e33d6abd"). InnerVolumeSpecName "kube-api-access-bt5cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:02.230852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.230744    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-dshm" pod "29455243-b7f4-4059-8066-ddf8e33d6abd" (UID: "29455243-b7f4-4059-8066-ddf8e33d6abd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.230852 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.230762    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29455243-b7f4-4059-8066-ddf8e33d6abd-tls-certs" pod "29455243-b7f4-4059-8066-ddf8e33d6abd" (UID: "29455243-b7f4-4059-8066-ddf8e33d6abd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:34:02.329916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.329887    5817 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29455243-b7f4-4059-8066-ddf8e33d6abd-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.329916 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.329915    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt5cf\" (UniqueName: \"kubernetes.io/projected/29455243-b7f4-4059-8066-ddf8e33d6abd-kube-api-access-bt5cf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.330130 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.329925    5817 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29455243-b7f4-4059-8066-ddf8e33d6abd-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.712808 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:02.712766    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:03.008123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.008039    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"
Nov 03 11:34:03.008123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.008061    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" event={"ID":"29455243-b7f4-4059-8066-ddf8e33d6abd","Type":"ContainerDied","Data":"7e839854773b5005691e1b5b3a3865ac44389ba23601d4f68cccd195ae4dde26"}
Nov 03 11:34:03.008123 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.008113    5817 scope.go:122] "RemoveContainer" containerID="9679a69ef07a09ee16b0884e36dbbd3c4ed8968703d0cc677742354244738756"
Nov 03 11:34:03.030063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.030038    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"]
Nov 03 11:34:03.036025 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.036005    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf"]
Nov 03 11:34:03.061243 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.061215    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-w3-b1c6f6-kserve-69d9568886-84nlf" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:34:03.427649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.427278    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"]
Nov 03 11:34:03.427649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.427608    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt" podUID="812be958-9740-4509-9b3a-39b8b1cebc50" containerName="maas-api" containerID="cri-o://cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787" gracePeriod=30
Nov 03 11:34:03.510863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.510823    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:03.658411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.658382    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:03.715823 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.715792    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 11:34:03.744521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.744487    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-metrics-certs\") pod \"812be958-9740-4509-9b3a-39b8b1cebc50\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") "
Nov 03 11:34:03.744747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.744611    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812be958-9740-4509-9b3a-39b8b1cebc50-kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/812be958-9740-4509-9b3a-39b8b1cebc50-kube-api-access-kflts\") pod \"812be958-9740-4509-9b3a-39b8b1cebc50\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") "
Nov 03 11:34:03.744747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.744679    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/812be958-9740-4509-9b3a-39b8b1cebc50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/812be958-9740-4509-9b3a-39b8b1cebc50-openshift-service-ca\") pod \"812be958-9740-4509-9b3a-39b8b1cebc50\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") "
Nov 03 11:34:03.744747 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.744713    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-maas-api-tls\") pod \"812be958-9740-4509-9b3a-39b8b1cebc50\" (UID: \"812be958-9740-4509-9b3a-39b8b1cebc50\") "
Nov 03 11:34:03.746359 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.746330    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812be958-9740-4509-9b3a-39b8b1cebc50-openshift-service-ca" pod "812be958-9740-4509-9b3a-39b8b1cebc50" (UID: "812be958-9740-4509-9b3a-39b8b1cebc50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:34:03.748094 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.748058    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-maas-api-tls" pod "812be958-9740-4509-9b3a-39b8b1cebc50" (UID: "812be958-9740-4509-9b3a-39b8b1cebc50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:34:03.749605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.748862    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-metrics-certs" pod "812be958-9740-4509-9b3a-39b8b1cebc50" (UID: "812be958-9740-4509-9b3a-39b8b1cebc50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:34:03.749605 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.748992    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812be958-9740-4509-9b3a-39b8b1cebc50-kube-api-access-kflts" pod "812be958-9740-4509-9b3a-39b8b1cebc50" (UID: "812be958-9740-4509-9b3a-39b8b1cebc50"). InnerVolumeSpecName "kube-api-access-kflts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:03.793964 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.793933    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:03.846514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.846489    5817 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:03.846514 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.846512    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/812be958-9740-4509-9b3a-39b8b1cebc50-kube-api-access-kflts\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:03.846724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.846523    5817 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/812be958-9740-4509-9b3a-39b8b1cebc50-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:03.846724 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.846533    5817 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/812be958-9740-4509-9b3a-39b8b1cebc50-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:03.900927 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.900900    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" path="/var/lib/kubelet/pods/29455243-b7f4-4059-8066-ddf8e33d6abd/volumes"
Nov 03 11:34:03.901375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:03.901361    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e3565d-d4a2-4515-b78d-237beff6a5cf" path="/var/lib/kubelet/pods/c6e3565d-d4a2-4515-b78d-237beff6a5cf/volumes"
Nov 03 11:34:04.027014 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.026928    5817 generic.go:356] "Generic (PLEG): container finished" podID="812be958-9740-4509-9b3a-39b8b1cebc50" containerID="cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787" exitCode=0
Nov 03 11:34:04.027464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.027108    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"
Nov 03 11:34:04.027643 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.027623    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt" event={"ID":"812be958-9740-4509-9b3a-39b8b1cebc50","Type":"ContainerDied","Data":"cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787"}
Nov 03 11:34:04.027738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.027691    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt" event={"ID":"812be958-9740-4509-9b3a-39b8b1cebc50","Type":"ContainerDied","Data":"2d653a90856e433a96f895fdfb45f2df9ce4e3b6f7ab8d71600a307ea09a4efa"}
Nov 03 11:34:04.027738 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.027717    5817 scope.go:122] "RemoveContainer" containerID="cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787"
Nov 03 11:34:04.040022 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.040004    5817 scope.go:122] "RemoveContainer" containerID="cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787"
Nov 03 11:34:04.040300 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:34:04.040278    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787\": container with ID starting with cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787 not found: ID does not exist" containerID="cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787"
Nov 03 11:34:04.040391 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.040305    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787"} err="failed to get container status \"cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787\": rpc error: code = NotFound desc = could not find container \"cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787\": container with ID starting with cba40b6d12c96e494f6fcb586b2c7a829a6859d994fae053892b9653b0793787 not found: ID does not exist"
Nov 03 11:34:04.057802 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.057779    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"]
Nov 03 11:34:04.063452 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.063426    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b1c6f6-7664465554-mkvvt"]
Nov 03 11:34:04.197128 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.197095    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:04.343840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.343748    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:04.499813 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:04.499778    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:05.142237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:05.142199    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-82pqb"]
Nov 03 11:34:05.386602 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:05.386544    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:05.901472 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:05.901430    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812be958-9740-4509-9b3a-39b8b1cebc50" path="/var/lib/kubelet/pods/812be958-9740-4509-9b3a-39b8b1cebc50/volumes"
Nov 03 11:34:09.002999 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:09.002964    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:09.592186 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:09.592147    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv" podUID="6b827954-c908-4603-998d-501f19de5a7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:34:09.592371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:09.592223    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9" podUID="f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:34:10.018549 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:10.018511    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:10.160362 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:10.160329    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:10.316659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:10.316525    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"]
Nov 03 11:34:10.317209 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:10.317174    5817 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" containerID="cri-o://1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83" gracePeriod=30
Nov 03 11:34:10.406844 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:10.406790    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:11.397670 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:11.397634    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:12.005644 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:12.005610    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vrn6v_cf5191a6-ee98-4b6b-b267-2af0e92b6f40/manager/0.log"
Nov 03 11:34:12.175082 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:12.175055    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jjvlt_d5049482-53a8-4070-8b53-47a3ce9cf145/kuadrant-console-plugin/0.log"
Nov 03 11:34:12.239336 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:12.239309    5817 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:34:12.239477 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:12.239358    5817 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:34:12.634969 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:12.634932    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-klrdr_63222367-1778-4565-9e4b-c2b33621751e/manager/0.log"
Nov 03 11:34:12.831510 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:12.831483    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-82pqb_6b444367-e539-4961-a151-0847a752f0b1/limitador/0.log"
Nov 03 11:34:15.573824 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.573795    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 11:34:15.658538 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658444    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-credential-socket\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.658538 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658509    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-kube-api-access-nh7fr\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-kube-api-access-nh7fr\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.658840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658689    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-podinfo\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.658840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658723    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-envoy\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.658840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658753    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-certs\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.658840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658776    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-credential-socket" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.658840 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658786    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-data\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.659136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658861    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-socket\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.659136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658907    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istiod-ca-cert\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.659136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.658936    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-token\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-token\") pod \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\" (UID: \"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1\") "
Nov 03 11:34:15.659136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.659041    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-certs" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.659136 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.659059    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-data" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.659448 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.659153    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-socket" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.659540 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.659523    5817 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.659652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.659546    5817 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.659652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.659562    5817 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.659652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.659609    5817 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.659652 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.659559    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istiod-ca-cert" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:34:15.661096 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.661060    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-envoy" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.661233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.661098    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-podinfo" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:34:15.661233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.661116    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-kube-api-access-nh7fr" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "kube-api-access-nh7fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:15.661622 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.661607    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-token" pod "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" (UID: "ea87f772-3cf2-44b7-9bfa-b72418e7a0a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:15.760250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.760216    5817 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.760250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.760245    5817 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.760470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.760260    5817 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.760470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.760273    5817 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.760470 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:15.760285    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh7fr\" (UniqueName: \"kubernetes.io/projected/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1-kube-api-access-nh7fr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:16.020860 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.020813    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"]
Nov 03 11:34:16.023223 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.023202    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72db4f18-nwhfg"]
Nov 03 11:34:16.113830 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.113800    5817 generic.go:356] "Generic (PLEG): container finished" podID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerID="1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83" exitCode=0
Nov 03 11:34:16.113977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.113840    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"
Nov 03 11:34:16.113977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.113857    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" event={"ID":"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1","Type":"ContainerDied","Data":"1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83"}
Nov 03 11:34:16.113977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.113902    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7" event={"ID":"ea87f772-3cf2-44b7-9bfa-b72418e7a0a1","Type":"ContainerDied","Data":"d4500c18997d9004baadb887328645bb99eedbf641ddf021a9fabd3dfebd9d34"}
Nov 03 11:34:16.113977 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.113923    5817 scope.go:122] "RemoveContainer" containerID="1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83"
Nov 03 11:34:16.124735 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.124714    5817 scope.go:122] "RemoveContainer" containerID="1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83"
Nov 03 11:34:16.124998 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:34:16.124981    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83\": container with ID starting with 1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83 not found: ID does not exist" containerID="1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83"
Nov 03 11:34:16.125063 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.125007    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83"} err="failed to get container status \"1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83\": rpc error: code = NotFound desc = could not find container \"1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83\": container with ID starting with 1cd3296400164c0fd7aa1ec73c099550e7ad83f3554b733d1a0bcd4e67085c83 not found: ID does not exist"
Nov 03 11:34:16.131410 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.131387    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"]
Nov 03 11:34:16.142008 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:16.138848    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jxdz7"]
Nov 03 11:34:17.495671 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.495646    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 11:34:17.529784 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.529763    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 11:34:17.577127 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.577102    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b827954-c908-4603-998d-501f19de5a7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b827954-c908-4603-998d-501f19de5a7a-plugins-config-volume\") pod \"6b827954-c908-4603-998d-501f19de5a7a\" (UID: \"6b827954-c908-4603-998d-501f19de5a7a\") "
Nov 03 11:34:17.577290 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.577207    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b827954-c908-4603-998d-501f19de5a7a-kube-api-access-246w5\" (UniqueName: \"kubernetes.io/projected/6b827954-c908-4603-998d-501f19de5a7a-kube-api-access-246w5\") pod \"6b827954-c908-4603-998d-501f19de5a7a\" (UID: \"6b827954-c908-4603-998d-501f19de5a7a\") "
Nov 03 11:34:17.577543 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.577519    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b827954-c908-4603-998d-501f19de5a7a-plugins-config-volume" pod "6b827954-c908-4603-998d-501f19de5a7a" (UID: "6b827954-c908-4603-998d-501f19de5a7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:34:17.579406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.579387    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b827954-c908-4603-998d-501f19de5a7a-kube-api-access-246w5" pod "6b827954-c908-4603-998d-501f19de5a7a" (UID: "6b827954-c908-4603-998d-501f19de5a7a"). InnerVolumeSpecName "kube-api-access-246w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:17.677939 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.677916    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-plugins-config-volume\") pod \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\" (UID: \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\") "
Nov 03 11:34:17.678085 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.677983    5817 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-kube-api-access-rh7g6\" (UniqueName: \"kubernetes.io/projected/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-kube-api-access-rh7g6\") pod \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\" (UID: \"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d\") "
Nov 03 11:34:17.678227 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.678202    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b827954-c908-4603-998d-501f19de5a7a-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:17.678337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.678228    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246w5\" (UniqueName: \"kubernetes.io/projected/6b827954-c908-4603-998d-501f19de5a7a-kube-api-access-246w5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:17.678337 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.678285    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-plugins-config-volume" pod "f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" (UID: "f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:34:17.680217 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.680199    5817 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-kube-api-access-rh7g6" pod "f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" (UID: "f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d"). InnerVolumeSpecName "kube-api-access-rh7g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:17.779397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.779339    5817 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh7g6\" (UniqueName: \"kubernetes.io/projected/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-kube-api-access-rh7g6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:17.779397 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.779362    5817 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:17.899668 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.899640    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62da97d-bfc8-4a9a-94c4-d87895bdcf5e" path="/var/lib/kubelet/pods/e62da97d-bfc8-4a9a-94c4-d87895bdcf5e/volumes"
Nov 03 11:34:17.900060 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:17.900046    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" path="/var/lib/kubelet/pods/ea87f772-3cf2-44b7-9bfa-b72418e7a0a1/volumes"
Nov 03 11:34:18.128936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.128873    5817 generic.go:356] "Generic (PLEG): container finished" podID="f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" containerID="dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9" exitCode=0
Nov 03 11:34:18.128936 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.128916    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"
Nov 03 11:34:18.129167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.128957    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9" event={"ID":"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d","Type":"ContainerDied","Data":"dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9"}
Nov 03 11:34:18.129167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.129019    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9" event={"ID":"f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d","Type":"ContainerDied","Data":"9fb17341791a734d873de88da63b817171661be09fc0324e79aadf1792dca58a"}
Nov 03 11:34:18.129167 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.129051    5817 scope.go:122] "RemoveContainer" containerID="dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9"
Nov 03 11:34:18.130480 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.130460    5817 generic.go:356] "Generic (PLEG): container finished" podID="6b827954-c908-4603-998d-501f19de5a7a" containerID="a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e" exitCode=0
Nov 03 11:34:18.130634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.130489    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv" event={"ID":"6b827954-c908-4603-998d-501f19de5a7a","Type":"ContainerDied","Data":"a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e"}
Nov 03 11:34:18.130634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.130508    5817 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"
Nov 03 11:34:18.130634 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.130534    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv" event={"ID":"6b827954-c908-4603-998d-501f19de5a7a","Type":"ContainerDied","Data":"1039d975d8f69b9ad7ff14707cb97e9d0c4e853816612601d9b3fba91f0c143c"}
Nov 03 11:34:18.139189 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.139173    5817 scope.go:122] "RemoveContainer" containerID="dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9"
Nov 03 11:34:18.139435 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:34:18.139417    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9\": container with ID starting with dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9 not found: ID does not exist" containerID="dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9"
Nov 03 11:34:18.139485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.139441    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9"} err="failed to get container status \"dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9\": rpc error: code = NotFound desc = could not find container \"dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9\": container with ID starting with dd73b213c13ef5b607aa0945db6686385d3d86c19a7c99ab4649a3d2fc06ecc9 not found: ID does not exist"
Nov 03 11:34:18.139485 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.139455    5817 scope.go:122] "RemoveContainer" containerID="a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e"
Nov 03 11:34:18.147383 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.147363    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"]
Nov 03 11:34:18.150744 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.150725    5817 scope.go:122] "RemoveContainer" containerID="a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e"
Nov 03 11:34:18.151004 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:34:18.150980    5817 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e\": container with ID starting with a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e not found: ID does not exist" containerID="a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e"
Nov 03 11:34:18.151104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.151010    5817 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e"} err="failed to get container status \"a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e\": rpc error: code = NotFound desc = could not find container \"a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e\": container with ID starting with a170211e1bb42f8d41732f025529c7e9f9410b312c2ae97a84667a1166626e9e not found: ID does not exist"
Nov 03 11:34:18.151323 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.151305    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b1c6f6-6cd67bc9f7-wnxfv"]
Nov 03 11:34:18.161625 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.161603    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"]
Nov 03 11:34:18.163748 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:18.163730    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b1c6f6-69d745bb99-lbjx9"]
Nov 03 11:34:19.899944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:19.899914    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b827954-c908-4603-998d-501f19de5a7a" path="/var/lib/kubelet/pods/6b827954-c908-4603-998d-501f19de5a7a/volumes"
Nov 03 11:34:19.900423 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:19.900265    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" path="/var/lib/kubelet/pods/f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d/volumes"
Nov 03 11:34:22.411941 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:34:22.411908    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:34:24.269859 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:24.269818    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:36.020862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:36.020824    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"]
Nov 03 11:34:36.022982 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:36.022962    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4179e8e-ng88v"]
Nov 03 11:34:36.351065 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:36.350987    5817 scope.go:122] "RemoveContainer" containerID="b18df56970b8d5d5e48f80a2b343689598140a0ebc3a5def68fd7c618fb378d5"
Nov 03 11:34:36.360421 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:36.360394    5817 scope.go:122] "RemoveContainer" containerID="a34f8e16e5975dd0ac5ddddf86543e4ddf81c44fe80b019e96a92869dbd259f9"
Nov 03 11:34:36.369109 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:36.369092    5817 scope.go:122] "RemoveContainer" containerID="cbeab8ab754b60c107f703258bc74731ad252f8696163c714475052b2cce3b1b"
Nov 03 11:34:36.377464 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:36.377449    5817 scope.go:122] "RemoveContainer" containerID="418542a31f529ec867401e84aef632fff628b302195fb6788c427429f1a79d81"
Nov 03 11:34:36.385378 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:36.385357    5817 scope.go:122] "RemoveContainer" containerID="e226edd649524463aaf66a235f6f4e282358fa8427324b4a953a698799f8a371"
Nov 03 11:34:36.393366 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:36.393351    5817 scope.go:122] "RemoveContainer" containerID="af2e028ceeec37a2091038669e3e2b1a53588df3cf196fbfc81fce881a0b8ab2"
Nov 03 11:34:37.647281 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:37.647250    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:37.821535 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:37.821501    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:37.899429 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:37.899355    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d617dc3-d6b3-4292-87a2-b8e6ff353ed1" path="/var/lib/kubelet/pods/8d617dc3-d6b3-4292-87a2-b8e6ff353ed1/volumes"
Nov 03 11:34:42.437108 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:42.437073    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:42.612053 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:42.612024    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:47.019277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:47.019235    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"]
Nov 03 11:34:47.023494 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:47.023466    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15242534-q2x5l"]
Nov 03 11:34:47.900206 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:47.900172    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84c9456-bce4-4d4b-aea1-43680d02755f" path="/var/lib/kubelet/pods/f84c9456-bce4-4d4b-aea1-43680d02755f/volumes"
Nov 03 11:34:50.621952 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:50.621920    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-00426132-lsk55_b812b1ad-4e6f-4e12-bdaf-4b1f70c2172a/revoke-keys/0.log"
Nov 03 11:34:50.796917 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:50.796890    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-00534afc-svm9x_1b130fdf-59ac-4f66-ad7c-2a04636e1dd3/revoke-keys/0.log"
Nov 03 11:34:50.971554 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:50.971530    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2115d230-h42wd_5de3163e-4bba-4211-99e7-3c32f6eb8d2d/revoke-keys/0.log"
Nov 03 11:34:51.149791 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:51.149764    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-451037cc-5psg7_4c418ec8-5cf4-4185-8fe0-5291f681684a/revoke-keys/0.log"
Nov 03 11:34:51.336923 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:51.336859    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-58f9cb61-xs5p8_72a2f80e-8e65-40a7-b522-d6f69edd16f9/revoke-keys/0.log"
Nov 03 11:34:51.704654 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:51.704627    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-710f62ef-q49bb_b29adfbc-029a-4730-beb4-2f7bb88030d5/revoke-keys/0.log"
Nov 03 11:34:51.903911 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:51.903880    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b68cb383-mr45g_e273cace-a356-4d49-bb5b-4b49b19251bf/revoke-keys/0.log"
Nov 03 11:34:52.411534 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:34:52.411500    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:34:52.686157 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:52.686130    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e71f2799-w7rcf_865effdb-6bf7-47ef-830a-b3c4ccdb4ea0/revoke-keys/0.log"
Nov 03 11:34:52.863343 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:52.863316    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fd6e98d4-svtns_f043817c-c3d3-4a48-a8eb-de7a02663aad/revoke-keys/0.log"
Nov 03 11:34:53.039372 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:53.039307    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff701a8c-p5lw8_56b42064-852e-4039-940b-79c5bbfdc133/revoke-keys/0.log"
Nov 03 11:34:55.112371 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:55.112342    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:34:55.147621 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:55.147596    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vrn6v_cf5191a6-ee98-4b6b-b267-2af0e92b6f40/manager/0.log"
Nov 03 11:34:55.328539 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:55.328509    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jjvlt_d5049482-53a8-4070-8b53-47a3ce9cf145/kuadrant-console-plugin/0.log"
Nov 03 11:34:55.708946 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:55.708909    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-klrdr_63222367-1778-4565-9e4b-c2b33621751e/manager/0.log"
Nov 03 11:34:55.903462 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:55.903432    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-82pqb_6b444367-e539-4961-a151-0847a752f0b1/limitador/0.log"
Nov 03 11:34:56.853898 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:56.853865    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fn6gpr_282aee57-34ba-4c1d-a305-13b93667a0cc/istio-proxy/0.log"
Nov 03 11:34:57.447328 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:57.447295    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb777df86-9cpw5_86d8e220-c327-4e65-95d6-a7394f5c1b9e/kube-auth-proxy/0.log"
Nov 03 11:34:58.242782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:58.242748    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-c7n4c_edf3ef4f-1840-4e68-9049-ec8a195d5e9b/istio-proxy/1.log"
Nov 03 11:34:59.164299 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:59.164264    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48_37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9/istio-proxy/0.log"
Nov 03 11:34:59.343039 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:59.343009    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xkl44_3023359d-37a2-4fbf-9241-442ecaf7fd1f/payload-pre-processing/0.log"
Nov 03 11:34:59.544089 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:34:59.544055    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2f7kn_99ada003-4afe-4019-8dab-6c9dadf4b888/payload-processing/0.log"
Nov 03 11:35:00.203267 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:00.203227    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-c7n4c_edf3ef4f-1840-4e68-9049-ec8a195d5e9b/istio-proxy/0.log"
Nov 03 11:35:00.933312 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:00.933276    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2xhzm_042afe48-f39e-4b28-bb65-d7db0e147638/main/0.log"
Nov 03 11:35:01.024650 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:01.024621    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"]
Nov 03 11:35:01.027117 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:01.027088    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451037cc-5psg7"]
Nov 03 11:35:01.299105 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:01.299026    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-x9t8q_550263cd-a336-4f27-abad-fafbbc1c61c6/main/0.log"
Nov 03 11:35:01.659030 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:01.658955    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-lrkj6_618e5bc7-d6bb-4234-80c5-a3da85ed8cb2/main/0.log"
Nov 03 11:35:01.900073 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:01.900045    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c418ec8-5cf4-4185-8fe0-5291f681684a" path="/var/lib/kubelet/pods/4c418ec8-5cf4-4185-8fe0-5291f681684a/volumes"
Nov 03 11:35:01.995489 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:01.995463    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9qjlw_0a480b85-cd03-4052-a431-4d092fa8600b/main/0.log"
Nov 03 11:35:06.497075 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:06.497035    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:07.185792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:07.185740    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:09.055059 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:09.055027    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:22.411385 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:35:22.411348    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:35:23.128179 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128144    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v48x6/must-gather-vhxbl"]
Nov 03 11:35:23.128542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128527    5817 cpu_manager.go:414] "containerMap: removing container" podUID="7155fb21-319e-47f0-8680-359ca1a613b0" containerName="payload-pre-processing"
Nov 03 11:35:23.128542 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128542    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7155fb21-319e-47f0-8680-359ca1a613b0" containerName="payload-pre-processing"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128558    5817 cpu_manager.go:414] "containerMap: removing container" podUID="812be958-9740-4509-9b3a-39b8b1cebc50" containerName="maas-api"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128563    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812be958-9740-4509-9b3a-39b8b1cebc50" containerName="maas-api"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128592    5817 cpu_manager.go:414] "containerMap: removing container" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128597    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128605    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" containerName="payload-pre-processing"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128610    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" containerName="payload-pre-processing"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128617    5817 cpu_manager.go:414] "containerMap: removing container" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128621    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128643    5817 cpu_manager.go:414] "containerMap: removing container" podUID="6b827954-c908-4603-998d-501f19de5a7a" containerName="payload-processing"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128647    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b827954-c908-4603-998d-501f19de5a7a" containerName="payload-processing"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128655    5817 cpu_manager.go:414] "containerMap: removing container" podUID="865effdb-6bf7-47ef-830a-b3c4ccdb4ea0" containerName="revoke-keys"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128661    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865effdb-6bf7-47ef-830a-b3c4ccdb4ea0" containerName="revoke-keys"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128668    5817 cpu_manager.go:414] "containerMap: removing container" podUID="f4d72517-c9c3-4c7c-a75d-86b42d6c644f" containerName="payload-processing"
Nov 03 11:35:23.128702 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128672    5817 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d72517-c9c3-4c7c-a75d-86b42d6c644f" containerName="payload-processing"
Nov 03 11:35:23.129174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128728    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b827954-c908-4603-998d-501f19de5a7a" containerName="payload-processing"
Nov 03 11:35:23.129174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128735    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="865effdb-6bf7-47ef-830a-b3c4ccdb4ea0" containerName="revoke-keys"
Nov 03 11:35:23.129174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128741    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d72517-c9c3-4c7c-a75d-86b42d6c644f" containerName="payload-processing"
Nov 03 11:35:23.129174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128749    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="7155fb21-319e-47f0-8680-359ca1a613b0" containerName="payload-pre-processing"
Nov 03 11:35:23.129174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128755    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="812be958-9740-4509-9b3a-39b8b1cebc50" containerName="maas-api"
Nov 03 11:35:23.129174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128760    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="29455243-b7f4-4059-8066-ddf8e33d6abd" containerName="main"
Nov 03 11:35:23.129174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128765    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f5eb5a-b57d-4aa8-9354-4a25bcd3e99d" containerName="payload-pre-processing"
Nov 03 11:35:23.129174 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.128769    5817 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea87f772-3cf2-44b7-9bfa-b72418e7a0a1" containerName="istio-proxy"
Nov 03 11:35:23.131944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.131929    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v48x6/must-gather-vhxbl"
Nov 03 11:35:23.139424 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.139375    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v48x6\"/\"kube-root-ca.crt\""
Nov 03 11:35:23.139513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.139442    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v48x6\"/\"default-dockercfg-25c4q\""
Nov 03 11:35:23.139513 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.139468    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v48x6\"/\"openshift-service-ca.crt\""
Nov 03 11:35:23.140778 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.140757    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v48x6/must-gather-vhxbl"]
Nov 03 11:35:23.194962 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.194936    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:23.222897 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.222868    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfb69/must-gather-plpg6"]
Nov 03 11:35:23.226440 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.226425    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfb69/must-gather-plpg6"
Nov 03 11:35:23.236351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.236329    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfb69/must-gather-plpg6"]
Nov 03 11:35:23.236715 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.236698    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pgh\" (UniqueName: \"kubernetes.io/projected/42b43c0d-f2a9-45bb-92e7-b40a3117a1a2-kube-api-access-d9pgh\") pod \"must-gather-vhxbl\" (UID: \"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2\") " pod="openshift-must-gather-v48x6/must-gather-vhxbl"
Nov 03 11:35:23.236789 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.236739    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42b43c0d-f2a9-45bb-92e7-b40a3117a1a2-must-gather-output\") pod \"must-gather-vhxbl\" (UID: \"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2\") " pod="openshift-must-gather-v48x6/must-gather-vhxbl"
Nov 03 11:35:23.238307 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.238285    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfb69\"/\"kube-root-ca.crt\""
Nov 03 11:35:23.238403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.238296    5817 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfb69\"/\"openshift-service-ca.crt\""
Nov 03 11:35:23.238403 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.238318    5817 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kfb69\"/\"default-dockercfg-pn76j\""
Nov 03 11:35:23.337251 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.337221    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:23.337406 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.337382    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8d4729-3509-4b78-b34e-cb95cf6f037b-must-gather-output\") pod \"must-gather-plpg6\" (UID: \"0f8d4729-3509-4b78-b34e-cb95cf6f037b\") " pod="openshift-must-gather-kfb69/must-gather-plpg6"
Nov 03 11:35:23.337469 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.337455    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbgk7\" (UniqueName: \"kubernetes.io/projected/0f8d4729-3509-4b78-b34e-cb95cf6f037b-kube-api-access-dbgk7\") pod \"must-gather-plpg6\" (UID: \"0f8d4729-3509-4b78-b34e-cb95cf6f037b\") " pod="openshift-must-gather-kfb69/must-gather-plpg6"
Nov 03 11:35:23.337519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.337486    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pgh\" (UniqueName: \"kubernetes.io/projected/42b43c0d-f2a9-45bb-92e7-b40a3117a1a2-kube-api-access-d9pgh\") pod \"must-gather-vhxbl\" (UID: \"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2\") " pod="openshift-must-gather-v48x6/must-gather-vhxbl"
Nov 03 11:35:23.337519 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.337513    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42b43c0d-f2a9-45bb-92e7-b40a3117a1a2-must-gather-output\") pod \"must-gather-vhxbl\" (UID: \"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2\") " pod="openshift-must-gather-v48x6/must-gather-vhxbl"
Nov 03 11:35:23.337891 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.337872    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42b43c0d-f2a9-45bb-92e7-b40a3117a1a2-must-gather-output\") pod \"must-gather-vhxbl\" (UID: \"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2\") " pod="openshift-must-gather-v48x6/must-gather-vhxbl"
Nov 03 11:35:23.346609 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.346582    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pgh\" (UniqueName: \"kubernetes.io/projected/42b43c0d-f2a9-45bb-92e7-b40a3117a1a2-kube-api-access-d9pgh\") pod \"must-gather-vhxbl\" (UID: \"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2\") " pod="openshift-must-gather-v48x6/must-gather-vhxbl"
Nov 03 11:35:23.438712 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.438683    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbgk7\" (UniqueName: \"kubernetes.io/projected/0f8d4729-3509-4b78-b34e-cb95cf6f037b-kube-api-access-dbgk7\") pod \"must-gather-plpg6\" (UID: \"0f8d4729-3509-4b78-b34e-cb95cf6f037b\") " pod="openshift-must-gather-kfb69/must-gather-plpg6"
Nov 03 11:35:23.439074 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.438820    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8d4729-3509-4b78-b34e-cb95cf6f037b-must-gather-output\") pod \"must-gather-plpg6\" (UID: \"0f8d4729-3509-4b78-b34e-cb95cf6f037b\") " pod="openshift-must-gather-kfb69/must-gather-plpg6"
Nov 03 11:35:23.439120 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.439107    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8d4729-3509-4b78-b34e-cb95cf6f037b-must-gather-output\") pod \"must-gather-plpg6\" (UID: \"0f8d4729-3509-4b78-b34e-cb95cf6f037b\") " pod="openshift-must-gather-kfb69/must-gather-plpg6"
Nov 03 11:35:23.447546 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.447529    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v48x6/must-gather-vhxbl"
Nov 03 11:35:23.448351 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.448335    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbgk7\" (UniqueName: \"kubernetes.io/projected/0f8d4729-3509-4b78-b34e-cb95cf6f037b-kube-api-access-dbgk7\") pod \"must-gather-plpg6\" (UID: \"0f8d4729-3509-4b78-b34e-cb95cf6f037b\") " pod="openshift-must-gather-kfb69/must-gather-plpg6"
Nov 03 11:35:23.540967 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.540934    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfb69/must-gather-plpg6"
Nov 03 11:35:23.726218 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.726188    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v48x6/must-gather-vhxbl"]
Nov 03 11:35:23.727760 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:35:23.727731    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42b43c0d_f2a9_45bb_92e7_b40a3117a1a2.slice/crio-2d75541c51a74c164c8f60bc10bd18e1205c39d5e96986e203cf439598cd7973 WatchSource:0}: Error finding container 2d75541c51a74c164c8f60bc10bd18e1205c39d5e96986e203cf439598cd7973: Status 404 returned error can't find the container with id 2d75541c51a74c164c8f60bc10bd18e1205c39d5e96986e203cf439598cd7973
Nov 03 11:35:23.729770 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.729751    5817 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:35:23.814294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:23.814272    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfb69/must-gather-plpg6"]
Nov 03 11:35:23.815712 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:35:23.815688    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f8d4729_3509_4b78_b34e_cb95cf6f037b.slice/crio-c568bd616eefba4453af5f37d93ef8762c625a1de0de9b854670e3f08b9d5c7a WatchSource:0}: Error finding container c568bd616eefba4453af5f37d93ef8762c625a1de0de9b854670e3f08b9d5c7a: Status 404 returned error can't find the container with id c568bd616eefba4453af5f37d93ef8762c625a1de0de9b854670e3f08b9d5c7a
Nov 03 11:35:24.296792 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:24.296748    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:24.572237 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:24.572154    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/must-gather-vhxbl" event={"ID":"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2","Type":"ContainerStarted","Data":"2d75541c51a74c164c8f60bc10bd18e1205c39d5e96986e203cf439598cd7973"}
Nov 03 11:35:24.573303 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:24.573280    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfb69/must-gather-plpg6" event={"ID":"0f8d4729-3509-4b78-b34e-cb95cf6f037b","Type":"ContainerStarted","Data":"c568bd616eefba4453af5f37d93ef8762c625a1de0de9b854670e3f08b9d5c7a"}
Nov 03 11:35:25.582753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:25.582718    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/must-gather-vhxbl" event={"ID":"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2","Type":"ContainerStarted","Data":"2cf771e19f644cf4a7119bfbacfb6acc9d97b594133a8622c4504196ec3301f5"}
Nov 03 11:35:25.582753 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:25.582761    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/must-gather-vhxbl" event={"ID":"42b43c0d-f2a9-45bb-92e7-b40a3117a1a2","Type":"ContainerStarted","Data":"3fec8d6239475af78e042bcd396989e1eaf7b3ebed95b6cf5279b11a307a8982"}
Nov 03 11:35:25.601910 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:25.601848    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v48x6/must-gather-vhxbl" podStartSLOduration=1.712012003 podStartE2EDuration="2.601828121s" totalImagesPullingTime="889.816118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:35:25.597668842 +0000 UTC m=+3955.803250904" watchObservedRunningTime="2026-11-03 11:35:25.601828121 +0000 UTC m=+3955.807410173"
Nov 03 11:35:26.892491 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:26.892423    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zwj2x_8e1ffb3b-e837-4cb6-b2b2-b2c55771e14e/global-pull-secret-syncer/0.log"
Nov 03 11:35:27.095594 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:27.094908    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log"
Nov 03 11:35:27.264876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:27.264846    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal_2cb181debb656c62bb70f1078166094d/haproxy/0.log"
Nov 03 11:35:28.044356 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:28.044275    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:28.182041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:28.181960    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:34.694153 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:34.694124    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vrn6v_cf5191a6-ee98-4b6b-b267-2af0e92b6f40/manager/0.log"
Nov 03 11:35:34.750454 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:34.750338    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jjvlt_d5049482-53a8-4070-8b53-47a3ce9cf145/kuadrant-console-plugin/0.log"
Nov 03 11:35:35.232523 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:35.230398    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-klrdr_63222367-1778-4565-9e4b-c2b33621751e/manager/0.log"
Nov 03 11:35:36.128503 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:36.128472    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-82pqb_6b444367-e539-4961-a151-0847a752f0b1/limitador/0.log"
Nov 03 11:35:36.475412 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:36.475378    5817 scope.go:122] "RemoveContainer" containerID="7582f3e91393159d7f753447e28e57b6008d3f05b3988ee38c0f2992d1a77249"
Nov 03 11:35:36.507862 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:36.506443    5817 scope.go:122] "RemoveContainer" containerID="5b0bb6ed5e2b063e4c0ddad8b9c721578081c4c7511b79fcbdcb55bb2ef19f1f"
Nov 03 11:35:36.539375 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:36.539297    5817 scope.go:122] "RemoveContainer" containerID="b3c6dd6ec053294a402cc4a5e4dc61c753cd3a8dc74346bb89eae3878c83d93e"
Nov 03 11:35:36.563810 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:36.558550    5817 scope.go:122] "RemoveContainer" containerID="1e7bc3ca0593f371b99ce3b8f33c0c68aa1a6d5e46d9cdb00362800f441dc734"
Nov 03 11:35:36.586772 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:36.586751    5817 scope.go:122] "RemoveContainer" containerID="792dfd0453877722da1674e611e0c6e71316fffa24f0f5cea086ec89375c5025"
Nov 03 11:35:36.636064 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:36.635360    5817 scope.go:122] "RemoveContainer" containerID="5014c892d5c64776bbc54743a7d4f36a0a53648786b5230a1d408b412ce545e9"
Nov 03 11:35:37.732250 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:37.732211    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xqq68_bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/kube-state-metrics/0.log"
Nov 03 11:35:37.760675 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:37.760599    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xqq68_bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/kube-rbac-proxy-main/0.log"
Nov 03 11:35:37.784361 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:37.784275    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xqq68_bd1c1f57-6b4b-4cd9-9245-0d71ef61b0f2/kube-rbac-proxy-self/0.log"
Nov 03 11:35:37.942398 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:37.942369    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpkfs_6aa3275d-c0c8-4ffb-9264-280b10d1672c/node-exporter/0.log"
Nov 03 11:35:37.964294 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:37.964265    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpkfs_6aa3275d-c0c8-4ffb-9264-280b10d1672c/kube-rbac-proxy/0.log"
Nov 03 11:35:37.987418 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:37.987338    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpkfs_6aa3275d-c0c8-4ffb-9264-280b10d1672c/init-textfile/0.log"
Nov 03 11:35:38.096595 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.094222    5817 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"]
Nov 03 11:35:38.114436 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.113745    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8lfdz_ac2fcf63-43d0-4759-9e77-edd3500ff388/kube-rbac-proxy-main/0.log"
Nov 03 11:35:38.139363 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.139333    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8lfdz_ac2fcf63-43d0-4759-9e77-edd3500ff388/kube-rbac-proxy-self/0.log"
Nov 03 11:35:38.159219 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.159189    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"]
Nov 03 11:35:38.159411 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.159284    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.163041 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.163014    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8lfdz_ac2fcf63-43d0-4759-9e77-edd3500ff388/openshift-state-metrics/0.log"
Nov 03 11:35:38.223545 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.223514    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce5df295-143c-4805-aafc-5f700c804cab/prometheus/0.log"
Nov 03 11:35:38.253611 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.253499    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce5df295-143c-4805-aafc-5f700c804cab/config-reloader/0.log"
Nov 03 11:35:38.277719 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.277647    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce5df295-143c-4805-aafc-5f700c804cab/thanos-sidecar/0.log"
Nov 03 11:35:38.301521 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.301443    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce5df295-143c-4805-aafc-5f700c804cab/kube-rbac-proxy-web/0.log"
Nov 03 11:35:38.332007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.326378    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-podres\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.332007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.329095    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-lib-modules\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.332007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.329138    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cl4p\" (UniqueName: \"kubernetes.io/projected/bce59fb5-0938-4cda-bc7e-607dc35e0d37-kube-api-access-2cl4p\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.332007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.329171    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-proc\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.332007 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.329227    5817 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-sys\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.336357 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.333947    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce5df295-143c-4805-aafc-5f700c804cab/kube-rbac-proxy/0.log"
Nov 03 11:35:38.358054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.358015    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce5df295-143c-4805-aafc-5f700c804cab/kube-rbac-proxy-thanos/0.log"
Nov 03 11:35:38.385983 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.385443    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce5df295-143c-4805-aafc-5f700c804cab/init-config-reloader/0.log"
Nov 03 11:35:38.430706 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430672    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-podres\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.430876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430718    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-lib-modules\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.430876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430742    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cl4p\" (UniqueName: \"kubernetes.io/projected/bce59fb5-0938-4cda-bc7e-607dc35e0d37-kube-api-access-2cl4p\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.430876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430773    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-proc\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.430876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430807    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-podres\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.430876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430821    5817 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-sys\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.430876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430864    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-sys\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.430876 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430865    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-proc\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.431211 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.430823    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bce59fb5-0938-4cda-bc7e-607dc35e0d37-lib-modules\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.438782 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.438750    5817 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cl4p\" (UniqueName: \"kubernetes.io/projected/bce59fb5-0938-4cda-bc7e-607dc35e0d37-kube-api-access-2cl4p\") pod \"perf-node-gather-daemonset-6x9rm\" (UID: \"bce59fb5-0938-4cda-bc7e-607dc35e0d37\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.538355 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.538270    5817 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:38.904387 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:38.904301    5817 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"]
Nov 03 11:35:38.909470 ip-10-0-138-233 kubenswrapper[5817]: W1103 11:35:38.908910    5817 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbce59fb5_0938_4cda_bc7e_607dc35e0d37.slice/crio-e0c308b473a554aec560ef06180a31b0c017788c208a814997ac9adf1b1f4750 WatchSource:0}: Error finding container e0c308b473a554aec560ef06180a31b0c017788c208a814997ac9adf1b1f4750: Status 404 returned error can't find the container with id e0c308b473a554aec560ef06180a31b0c017788c208a814997ac9adf1b1f4750
Nov 03 11:35:39.815649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:39.814936    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm" event={"ID":"bce59fb5-0938-4cda-bc7e-607dc35e0d37","Type":"ContainerStarted","Data":"ffbe32468cd1c91f2832342cc387c9b64dd0febe4b22db2db12781e338cff9a2"}
Nov 03 11:35:39.815649 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:39.815001    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm" event={"ID":"bce59fb5-0938-4cda-bc7e-607dc35e0d37","Type":"ContainerStarted","Data":"e0c308b473a554aec560ef06180a31b0c017788c208a814997ac9adf1b1f4750"}
Nov 03 11:35:39.816024 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:39.815969    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:39.839619 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:39.837157    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm" podStartSLOduration=1.8371422480000001 podStartE2EDuration="1.837142248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:35:39.832962242 +0000 UTC m=+3970.038544298" watchObservedRunningTime="2026-11-03 11:35:39.837142248 +0000 UTC m=+3970.042724300"
Nov 03 11:35:40.834442 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:40.834405    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfb69/must-gather-plpg6" event={"ID":"0f8d4729-3509-4b78-b34e-cb95cf6f037b","Type":"ContainerStarted","Data":"f5f6cdba65a9f0b75d86bd713ffcb490e5e21cf377f8f2db14c371945f611889"}
Nov 03 11:35:40.835002 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:40.834457    5817 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfb69/must-gather-plpg6" event={"ID":"0f8d4729-3509-4b78-b34e-cb95cf6f037b","Type":"ContainerStarted","Data":"6852a232b7c686887e737abfdd426c45aaa61a1e578a1c1a9686c9b381499a52"}
Nov 03 11:35:43.648894 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:43.648862    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wtrwm_d613c8d4-0ecb-4178-839a-e626bd2646e5/dns/0.log"
Nov 03 11:35:43.674233 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:43.674204    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wtrwm_d613c8d4-0ecb-4178-839a-e626bd2646e5/kube-rbac-proxy/0.log"
Nov 03 11:35:43.705315 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:43.705268    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9bmpc_565f923e-a384-496e-be31-f99ddaec54ce/dns-node-resolver/0.log"
Nov 03 11:35:44.636405 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:44.636370    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxvr7_3bf53482-bc2a-4f95-a0ec-d2064baeb4f6/node-ca/0.log"
Nov 03 11:35:45.903796 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:45.903765    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fn6gpr_282aee57-34ba-4c1d-a305-13b93667a0cc/istio-proxy/0.log"
Nov 03 11:35:46.859907 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:46.859872    5817 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6x9rm"
Nov 03 11:35:46.881054 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:46.880993    5817 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kfb69/must-gather-plpg6" podStartSLOduration=7.676763471 podStartE2EDuration="23.880973728s" totalImagesPullingTime="16.204210257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:35:40.854296759 +0000 UTC m=+3971.059878811" watchObservedRunningTime="2026-11-03 11:35:46.880973728 +0000 UTC m=+3977.086555783"
Nov 03 11:35:48.042761 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:48.042293    5817 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"]
Nov 03 11:35:48.047731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:48.047703    5817 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00534afc-svm9x"]
Nov 03 11:35:49.918277 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:49.918242    5817 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b130fdf-59ac-4f66-ad7c-2a04636e1dd3" path="/var/lib/kubelet/pods/1b130fdf-59ac-4f66-ad7c-2a04636e1dd3/volumes"
Nov 03 11:35:50.789697 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:50.789660    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fn6gpr_282aee57-34ba-4c1d-a305-13b93667a0cc/istio-proxy/0.log"
Nov 03 11:35:51.334247 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:51.334208    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb777df86-9cpw5_86d8e220-c327-4e65-95d6-a7394f5c1b9e/kube-auth-proxy/0.log"
Nov 03 11:35:51.489266 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:51.489233    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-c7n4c_edf3ef4f-1840-4e68-9049-ec8a195d5e9b/istio-proxy/0.log"
Nov 03 11:35:51.492104 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:51.492016    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-c7n4c_edf3ef4f-1840-4e68-9049-ec8a195d5e9b/istio-proxy/1.log"
Nov 03 11:35:51.677561 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:51.677479    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48_37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9/istio-proxy/0.log"
Nov 03 11:35:51.707566 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:51.707539    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xkl44_3023359d-37a2-4fbf-9241-442ecaf7fd1f/payload-pre-processing/0.log"
Nov 03 11:35:51.747944 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:51.747908    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2f7kn_99ada003-4afe-4019-8dab-6c9dadf4b888/payload-processing/0.log"
Nov 03 11:35:52.445716 ip-10-0-138-233 kubenswrapper[5817]: E1103 11:35:52.445680    5817 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jq8jl" containerName="extractor"
Nov 03 11:35:54.750696 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:54.750659    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d9x57_254ba5e2-5f70-4980-9f35-705260e6b4a8/serve-healthcheck-canary/0.log"
Nov 03 11:35:54.899173 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:54.899141    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fn6gpr_282aee57-34ba-4c1d-a305-13b93667a0cc/istio-proxy/0.log"
Nov 03 11:35:54.912794 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:54.912757    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb777df86-9cpw5_86d8e220-c327-4e65-95d6-a7394f5c1b9e/kube-auth-proxy/0.log"
Nov 03 11:35:54.996182 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:54.996137    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-c7n4c_edf3ef4f-1840-4e68-9049-ec8a195d5e9b/istio-proxy/0.log"
Nov 03 11:35:54.997863 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:54.997829    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-c7n4c_edf3ef4f-1840-4e68-9049-ec8a195d5e9b/istio-proxy/1.log"
Nov 03 11:35:55.099659 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:55.098705    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-n5s48_37ef3ba8-5695-4f8f-9b6f-6b836b52f7e9/istio-proxy/0.log"
Nov 03 11:35:55.121241 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:55.121208    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xkl44_3023359d-37a2-4fbf-9241-442ecaf7fd1f/payload-pre-processing/0.log"
Nov 03 11:35:55.140617 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:55.140566    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2f7kn_99ada003-4afe-4019-8dab-6c9dadf4b888/payload-processing/0.log"
Nov 03 11:35:55.304534 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:55.304465    5817 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v4lr_0174417f-2cb0-4933-b9cc-7b044e2e8254/konnectivity-agent/0.log" limit=1024
Nov 03 11:35:55.583592 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:55.583041    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jq8jl_487f4942-ea5d-44a2-92dc-9fed886c9fdf/kube-rbac-proxy/0.log"
Nov 03 11:35:55.609812 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:55.609779    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jq8jl_487f4942-ea5d-44a2-92dc-9fed886c9fdf/exporter/0.log"
Nov 03 11:35:55.636731 ip-10-0-138-233 kubenswrapper[5817]: I1103 11:35:55.636700    5817 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jq8jl_487f4942-ea5d-44a2-92dc-9fed886c9fdf/extractor/0.log"